Letsencrypt SSL Certs

Introduction

We chose to use Letsencrypt SSL certificates in addition to the Cloudflare origin certs because we occasionally need to serve SSL websites and services directly and while the Cloudflare origin certs work well and are easy to setup, they are not publicly recognised. To set up the Certbot service to download the SSL certs on Raisin with Cloudflare DNS, we will want to use a wildcard cert so that we don't have to keep on adding to a growing list of domain names

Installation

To set up the certbot Let's Encrypt service specifically using cert bot on the reverse proxy, Raisin, with Cloudflare DNS. Since we are using a wildcard, the DNS-01 challenge is the only way to go, and using the Cloudflare API makes it completely automated. The reason for choosing Raisin for the cert download is that it will nearly always need to be used on Raisin as it is the host that Pfsense directs all https traffic to.

Install certbot

This setup ensures we get our wildcard certificates without needing a webserver to be reachable from the internet, keeping our security tight.

🏛️ Phase 1: Clean Up & Core Install

Ubuntu 24.04 (Noble) likes its packages to be "clean." Mixing apt and snap is the most common way to break SSL renewals. To purge any old apt versions:

sudo apt-get remove certbot

Install the Snap core (the "engine" for snaps)

sudo snap install core; sudo snap refresh core

Create the Symlink: (This ensures when we type certbot, the system actually finds the snap version.)

sudo ln -s /snap/bin/certbot /usr/bin/certbot

🔑Cloudflare API Token

To get the Cloudflare API Token, we need to head into the Cloudflare dashboard. Since we have both .net and .uk domains, we can actually handle them with a single command or separate ones depending on whether we want one "combined" certificate or two distinct ones. in our case we will get two separate keys

• Log in to your Cloudflare Dashboard.
• Click on the User Profile icon (top right) and select My Profile.
• Click API Tokens in the left sidebar.
• Click Create Token.
• Find the template "Edit zone DNS" and click Use template.
• Permissions: Ensure it says Zone - DNS - Edit (and optionally Zone - Zone - Read).
• Zone Resources: * If you want one token for both domains, select Include - All zones.
  • If you want to be extra secure, select Include - Specific zone and add both seaoffate.net and seaoffate.uk.
• Click Continue to summary and then Create Token.
• Copy the token immediately! Cloudflare won't show it to you again.

🔑 Phase 3: The Cloudflare Plugin & Secret

Since we want *.seaoffate.net and *.seaoffate.net Certbot needs to "talk" to Cloudflare to create a temporary TXT record. We have two choices here based on your cloudflare_net.ini and cloudflare_uk.ini setup. If we had wanted a single certificate that covers both domains (ideal for a single load balancer like Raisin), we could have used just one .ini file as long as the API token inside has permission for both zones. However, we prefer to keep them separate so we just ran the command twice and have two separate files. Certbot is smart enough to create two separate renewal profiles in /etc/letsencrypt/renewal/.

• To Authorize the plugin

sudo snap set certbot trust-plugin-with-root=ok

Install the Cloudflare DNS plugin

sudo snap install certbot-dns-cloudflare

Create the Credentials File It's best practice to keep this in the /etc/letsencrypt directory, note we also have a cert for the .uk

sudo mkdir -p /etc/letsencrypt/cloudflare
sudo nano /etc/letsencrypt/cloudflare/cloudflare_net.ini

Add your API Token Note: Use a Token, not your Global Key. It only needs "Zone:DNS:Edit" permissions. we repeat this step to get the .uk key

dns_cloudflare_api_token = 0123456789abcdefyourtokenhere 

Then the same for the .uk key

sudo nano /etc/letsencrypt/cloudflare/cloudflare_uk.ini  

and copy the other key for the .uk domain

dns_cloudflare_api_token = 0123456789abcdefyourtokenhere

Lock down the files (Safety First!)

sudo chmod 600 /etc/letsencrypt/cloudflare/cloudflare_net.ini
sudo chmod 600 /etc/letsencrypt/cloudflare/cloudflare_uk.ini

🚀 Phase 4: Getting the First Certificate

Now we run the big command. This generates the cert and registers the Deploy Hook (so Nginx reloads itself every 60 days). We have chosen to keep the two separate files so we run the command twice. Certbot is smart enough to create two separate renewal profiles in /etc/letsencrypt/renewal/

sudo certbot certonly --dns-cloudflare \
 --dns-cloudflare-credentials /etc/letsencrypt/cloudflare/cloudflare_net.ini \
 --dns-cloudflare-propagation-seconds 60 \
 --deploy-hook "systemctl reload nginx" \
 --cert-name seaoffate.net \
 -d seaoffate.net -d "*.seaoffate.net"

sudo certbot certonly --dns-cloudflare \
 --dns-cloudflare-credentials /etc/letsencrypt/cloudflare/cloudflare_uk.ini \
 --dns-cloudflare-propagation-seconds 60 \
 --deploy-hook "systemctl reload nginx" \
 --cert-name seaoffate.uk \
 -d seaoffate.uk -d "*.seaoffate.uk"

🏛️ Phase 4: Verification

Always verify your work before closing the terminal. To Test the Renewal process:

sudo certbot renew --dry-run

Verify the Systemd Timer:

systemctl list-timers | grep certbot

Now that Raisin will fetch a new SSL certificate every time it becomes due we can look at deploying it to other VMs in which case the local PCs will also be using the SSL cert and not give security warnings.

🏛️ The Architecture: Push Distribution

Since we now have Raisin acting as the central certificate hub, the most professional way to handle this is to have Raisin "push" the certificates to Plum, Fig, Blackberry, and Quince the moment they are renewed. This avoids having to remember to do it manually and ensures all our "Fruits" stay in sync with the latest wildcard SSL. We will use a Bash script combined with SSH keys and Certbot's deploy-hook.

Establish SSH Trust (Passwordless Login)

For Raisin to push files, it needs to log into the other servers without being prompted for a password. On Raisin, check for a root SSH key:

sudo ls /root/.ssh/id_rsa.pub
# If it doesn't exist, create it: sudo ssh-keygen -t rsa -b 4096 -N ""

Copy the key to each destination server (repeat for all):

sudo ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
sudo ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
sudo ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
sudo ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
sudo ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
sudo ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]

🏛️ Final Prep: The "Destination" Folders. Before we run the script, we need to make sure the target directories exist on each "Fruit" and that nigel has the right permissions to drop files thereon each server run the following commands

sudo mkdir -p /etc/nginx/ssl/seaoffate.net
sudo chown nigel:nigel /etc/nginx/ssl/seaoffate.net
sudo mkdir -p /etc/nginx/ssl/seaoffate.uk
sudo chown nigel:nigel /etc/nginx/ssl/seaoffate.uk

Note: If a server doesn't have Nginx installed yet, the directory /etc/nginx might not exist. You can still create it manually.

🚀 The "Fruit-Sync" Master Script

Now, back on Raisin, let’s create the script that will automate the distribution. Create the script file:

sudo nano /usr/local/bin/deploy-wildcard.sh

Paste this code (Optimized for your specific setup):

#!/bin/bash
# Raisin SSL Distribution Bot - Dual Domain (.net & .uk)
# Optimized for Docker (Blackberry & Mango)
#
echo "--- Starting SSL Distribution: $(date) ---"
#
# 1. Reload local Nginx on Raisin (the source)
systemctl reload nginx
# 1.1 setup email notification 
SEND_TO="[email protected]"
#
send_notification() {
    local SUBJECT=$1
    local MESSAGE=$2
    echo "$MESSAGE" | mail -s "$SUBJECT" "$SEND_TO"
}
# 2. Destination Servers
SERVERS=("plum.seaoffate.local" "fig.seaoffate.local" "blackberry.seaoffate.local" "quince.seaoffate.local" "tayberry.seaoffate.local" "mango.seaoffate.local")
#
# 3. Source Cert Paths
NET_CERT_DIR="/etc/letsencrypt/live/seaoffate.net"
UK_CERT_DIR="/etc/letsencrypt/live/seaoffate.uk"
#
for SERVER in "${SERVERS[@]}"; do
    echo "--------------------------------------------"
    echo "Processing $SERVER..."
    #
    # PRE-CHECK: Get timestamp of current cert on destination (to see if it actually changes)
    # If file doesn't exist, we use '0'
    OLD_TS=$(ssh nigel@$SERVER "stat -c %Y /etc/nginx/ssl/seaoffate.net/fullchain.pem 2>/dev/null || echo 0")
    #
    # 4. Sync seaoffate.net
    echo "  Pushing .net certs..."
    scp "$NET_CERT_DIR/fullchain.pem" "nigel@$SERVER:/etc/nginx/ssl/seaoffate.net/fullchain.pem"
    scp "$NET_CERT_DIR/privkey.pem" "nigel@$SERVER:/etc/nginx/ssl/seaoffate.net/privkey.pem"
    #
    # 5. Sync seaoffate.uk
    echo "  Pushing .uk certs..."
    scp "$UK_CERT_DIR/fullchain.pem" "nigel@$SERVER:/etc/nginx/ssl/seaoffate.uk/fullchain.pem"
    scp "$UK_CERT_DIR/privkey.pem" "nigel@$SERVER:/etc/nginx/ssl/seaoffate.uk/privkey.pem"
    # POST-CHECK: Get new timestamp
    NEW_TS=$(ssh nigel@$SERVER "stat -c %Y /etc/nginx/ssl/seaoffate.net/fullchain.pem")
    #
    # 6. Execute reloads/restarts on the remote server
    ssh nigel@$SERVER "
        # A. Reload standard webservers (Apache/Nginx)
        if systemctl is-active --quiet apache2; then 
            sudo systemctl reload apache2 && echo '  Apache reloaded.'
        elif systemctl is-active --quiet nginx; then 
            sudo systemctl reload nginx && echo '  Nginx reloaded.'
        fi
        #
        # B. Docker Restarts (Only if cert has been updated)
        if [ \"$NEW_TS\" -gt \"$OLD_TS\" ]; then
            echo '  [!] Certificate update detected. Checking Docker containers...'
            #
            # Restart Dashy (Blackberry)
            if [ \$(docker ps -q -f name=dashy) ]; then
                echo '  Restarting Dashy...'
                docker restart dashy
            fi
            #
            # Restart Vaultwarden (Mango)
            if [ \$(docker ps -q -f name=vaultwarden) ]; then
                echo '  Restarting Vaultwarden...'
                docker restart vaultwarden
            fi
        else
            echo '  [.] No certificate change. Skipping Docker restarts.'
        fi
    "
    #
    if [ $? -eq 0 ]; then
        echo "Successfully processed $SERVER."
    else
        echo "ERROR: Issue communicating with $SERVER."
    fi
done
#
echo "--- SSL Distribution Complete: $(date) ---"
send_notif "✅ SSL Renewed & Distributed" "Certificates for .net and .uk have been pushed to all nodes and Nginx has been reloaded. Expiry is now approximately 90 days from today."

The script reloads Apache when the cert is delivered but the sudo systemctl reload apache2 requires a password that the script cannot give so we must add the no password rule to the sodoers file. on each target system (Plum, Fig, Blackberry and Quince) we must open the visudo file with the command

sudo visudo

Scroll down to the bottom and add the following line at the very bottom

nigel ALL=(ALL) NOPASSWD: /usr/bin/systemctl reload apache2, /usr/bin/systemctl reload nginx

This will allow systemctl reload apache2 to be run without adding a password

Automation Hook

To ensure the script runs every time a certificate is successfully renewed, use the reconfigure command on Raisin:

sudo certbot reconfigure --cert-name seaoffate.net --deploy-hook /usr/local/bin/deploy-wildcard.sh

Cert Delivery Failure Warning

If there is no updated certificate delivered we want to get a warning message some time before the current cert expires. As the certbot is supposed to fetch a new cert 30 days before the current cert expires it is reasonable that we should get an email 15 days before the expiry date. Further, it is possible that there could be something wrong with the redistribution scripts as well as a possible failure in the collection from Letsencrypt the best place to send the warning message from is Plum as it is one of the webservers that will need an up to date SSL cert. To that end we will have a script running every morning that simply checks the expiry date of it's SSL cert and if it less than 15 days to go it will send an email with a warning. the script needs to be created on plum so ssh to plum and create a file with :

sudo nano /usr/local/bin/check-ssl-expiry.sh

copy and paste the following

#!/bin/bash
# /usr/local/bin/check-ssl-expiry.sh on Plum
# Checks the actual LIVE certificates on the web
#
EMAIL="[email protected]"
THRESHOLD_DAYS=15
SITES=("seaoffate.net" "seaoffate.uk")
#
for SITE in "${SITES[@]}"; do
    # Use openssl to check the remote certificate expiry
    # This works even if the local files are fine but Nginx didn't reload
    EXPIRY_DATE=$(echo | openssl s_client -servername "$SITE" -connect "$SITE":443 2>/dev/null | openssl x509 -noout -enddate | cut -d= -f2)
    #
    if [ -z "$EXPIRY_DATE" ]; then
        echo "CRITICAL: Could not connect to $SITE to check SSL!" | mail -s "🚨 SSL CHECK FAILURE: $SITE" "$EMAIL"
        continue
    fi
    #
    # Convert dates to seconds for comparison
    EXPIRY_SECS=$(date -d "$EXPIRY_DATE" +%s)
    NOW_SECS=$(date +%s)
    DAYS_LEFT=$(( (EXPIRY_SECS - NOW_SECS) / 86400 ))
    #
    if [ "$DAYS_LEFT" -le "$THRESHOLD_DAYS" ]; then
        echo "WARNING: The SSL certificate for $SITE expires in $DAYS_LEFT days. Auto-renewal on Raisin may have failed!" | mail -s "⚠️ SSL EXPIRY WARNING: $SITE" "$EMAIL"
    fi
done   

make it executable with:

sudo chmod +x /usr/local/bin/check-ssl-expiry.sh

set it to run everyday at 08:30

sudo crontab -e

copy in the following at the end of the crontab

30 8 * * * /usr/local/bin/check-ssl-expiry.sh

Docker Volume Mapping (For Blackberry & Quince)

Since these nodes run on Docker, the certificates sitting on the host OS don't help the containers unless they are "mapped" in.

Using Certs in Docker On Docker nodes like Quince, do not install Apache. Instead, map the certificates into your docker-compose.yaml volumes: volumes:

 - /etc/nginx/ssl/seaoffate.net/fullchain.pem:/etc/ssl/certs/fullchain.pem:ro
 - /etc/nginx/ssl/seaoffate.net/privkey.pem:/etc/ssl/private/privkey.pem:ro

Changing user

While using the nigel account works ok and is quick to deploy, it would be better if there was a dedicated user that only had permission to download and deploy the certificates. That will be a job for another day but it should be don so that we can separate responsibilities in a more secure manner.