Letsencrypt Wildcard SSL Cert

Introduction

The time has come when we need to have SSL certs that can be used unrepentantly of Cloudflare. Cloudflare's proxy for webservers does work well but on the free tier they only listen to 443 and anything else will be rejected and return an invalid response. For some services like VPN (either OpenVPN or Wireguard) it does not matter as they will do their own encryption. However, services like Jellyfin use alternate ports for both http and https traffic so cannot be proxied by Cloudflare, to date it has not really mattered because streaming to remote locations is not really required too much and if it is we can use a VPN link if privacy is an issue. It would be nice to do Jellyfin over https but not really required. Enter N8N, it also works on non standard ports and does have a SSL engine that should work with Cloudflare but it does not appear to be as easy to setup as first thought and any outside access will certainly need to be secured.

Acquire the Wildcard SSL Certificate and key from Letsencrypt

Initial features to be recognised

The requirement is to have one key and certificate pair for the entire *.seaoffate.net and seaoffate.net DNS name. First it should be noted that the wildcard SSL cert will only cover one subdomain deep IE it will be ok for wiki.seaoffate.net or files.seaoffate.net but it will not be any good for any deeper like other.wiki.seaoffate.net. If any deeper nesting of DNS subdomain names is required they will have to be on a separate request. It is not likely to be much of a problem for seaoffate.net but in a business setting it could be an issue.

The next thing to be considered is that the certs will only be valid for a set period of time then they will expire, they would still do the encryption but would return errors to the browser due to the expiry date, the solution is to have a bot retrieve new certificates before the old cert expires and indeed there are Certbots especially setup for this purpose.

Another feature that will have to be addressed is that Letsencrypt will need to verify that seaoffate is really being hosted at the correct address. If it was just a one off cert then there is a process where the request is made to Letsencrypt through a Certbot and Letsencrypt return a string that needs to be added to the DNS provider, in this case Cloudflare, and as the text file is published along with the rest of the DNS information Letsencrypt can be sure that the user is indeed the owner of the domain (several text strings may need to be published for verification). This manual process is not too difficult or even time consuming but it is a manual process so it would be better if it can be automated. Fortunately and unsurprisingly, Cloudflare have a solution to the automating domain ownership problem in the form of API keys. Obviously it would be better if the API keys were kept private as any leak would allow a hostile actor to impersonate my websites forever.

This will be a wildcard certificate so it can be used by anything.seaoffate.net and as many times as required. It could be copied to all of the webservers and used in place of the existing Cloudflare origin cert (cloudflare full strict only requires that it be a cert that cloudflare can recognise and as this will be a public cert it will work), this would make the warnings when using webservers directly disappear but it is not certain that this is even desired let alone required.

Install and setup Certbot

Raisin Installation

We need to be logged in to Raisin for this part.

First of all if there is a second attempt to install certbot on Raisin or if it is just changing things around it would be better to remove any old installation. Then install a fresh version again

sudo apt update
sudo apt remove --purge certbot python3-certbot-nginx 2>/dev/null || true # Remove old Certbot
sudo rm -rf /etc/letsencrypt/ # Clean up old certs and configs
sudo apt install -y snapd
sudo snap install --classic certbot
sudo snap install certbot-dns-cloudflare
sudo ln -s /snap/bin/certbot /usr/bin/certbot # Ensure certbot command is available

We should create dedicated system user for deployment (certbot-deploy)

sudo useradd --no-create-home --shell /bin/false certbot-deploy

and generate SSh key for certbot-deploy and as the user has no home dir Keys will be stored in a secure location under /etc/letsencrypt/:

sudo mkdir -p /etc/letsencrypt/certbot-deploy/.ssh/
sudo chown certbot-deploy:certbot-deploy /etc/letsencrypt/certbot-deploy/.ssh/
sudo chmod 700 /etc/letsencrypt/certbot-deploy/.ssh/
sudo -u certbot-deploy ssh-keygen -t rsa -b 4096 -f /etc/letsencrypt/certbot-deploy/.ssh/id_rsa -N ""

Grant certbot-deploy sudo Access for ssh/scp on raisin to allow certbot-deploy to use ssh and scp commands without a password.

sudo visudo -f /etc/sudoers.d/certbot-deploy

Add the following content to the sudoers file

# Allow certbot-deploy user to execute ssh and scp without password
certbot-deploy ALL=(ALL) NOPASSWD: /usr/bin/ssh
certbot-deploy ALL=(ALL) NOPASSWD: /usr/bin/scp

Save and exit. make sure that the system allows a clean save and does not give options to edit or save indicating there is a syntax error in the commands.

Lychee Installation

We need to be logged in to Lychee for this part

Add nigel user to ssl-cert group (if not already): (To ensure nigel can read the letsencrypt_central directory)

sudo usermod -aG ssl-cert nigel

We need to Log out and log back in to Lychee for group changes to take effect. Next we create helper scripts for sudoers (chown -R and chmod -R) These scripts bypass sudoers' sensitivity to recursive commands. The two scripts are chown_certs_dir.sh and chmod_certs_dir.sh

sudo nano /usr/local/sbin/chown_certs_dir.sh

with the contents

#!/bin/bash ; 
/usr/bin/chown -R nigel:nigel "$1"

save and exit and set permissions and ownership

sudo chmod 700 /usr/local/sbin/chown_certs_dir.sh
sudo chown root:root /usr/local/sbin/chown_certs_dir.sh

and the second script is created with

sudo nano /usr/local/sbin/chmod_certs_dir.sh

with the contents

#!/bin/bash ; 
/usr/bin/chmod -R 700 "$1"

save and exit and set permissions and ownership

sudo chmod 700 /usr/local/sbin/chmod_certs_dir.sh
sudo chown root:root /usr/local/sbin/chmod_certs_dir.sh

We will need to configure nigel's sudoers on Lychee for NOPASSWD access

sudo visudo -f /etc/sudoers.d/nigel_cert_push

with the following contents

# Allow nigel to run specific commands for cert push via SSH without password
# For directory creation
nigel ALL=(ALL) NOPASSWD: /bin/mkdir -p /home/nigel/npm/letsencrypt/live/seaoffate.net

# For chown -R and chmod -R using helper scripts
nigel ALL=(ALL) NOPASSWD: /usr/local/sbin/chown_certs_dir.sh
nigel ALL=(ALL) NOPASSWD: /usr/local/sbin/chmod_certs_dir.sh

# For chmod 644 for fullchain.pem
nigel ALL=(ALL) NOPASSWD: /bin/bash -c "/bin/chmod 644 /home/nigel/npm/letsencrypt/live/seaoffate.net/fullchain.pem"

# For chmod 600 for privkey.pem
nigel ALL=(ALL) NOPASSWD: /bin/bash -c "/bin/chmod 600 /home/nigel/npm/letsencrypt/live/seaoffate.net/privkey.pem"

# For Nginx reload
nigel ALL=(ALL) NOPASSWD: /usr/bin/systemctl reload nginx
nigel ALL=(ALL) NOPASSWD: /usr/sbin/nginx -s reload

# For scp (client-side for receiving files)
nigel ALL=(ALL) NOPASSWD: /usr/bin/scp

Save and exit. Note: if the file does not save cleanly the syntax errors will need to be fixed.

Obtain Let's Encrypt Wildcard Certificates (on Raisin)

We need to be logged on to Raisin again for this part

We need to login to the Cloudflare control panel and go to the Profile section (from the username) then select "API Tokens" from the menu on the left. In the API Tokens main panel there should be a button "create Token"