Letsencrypt Wildcard SSL Cert

Introduction

The time has come when we need to have SSL certs that can be used independently of Cloudflare. Cloudflare's proxy for webservers does work well but on the free tier they only listen to 443 and anything else will be rejected and return an invalid response. For some services like VPN (either OpenVPN or Wireguard) it does not matter as they will do their own encryption. However, services like Jellyfin use alternate ports for both http and https traffic so cannot be proxied by Cloudflare, to date it has not really mattered because streaming to remote locations is not really required too much and if it is we can use a VPN link if privacy is an issue. It would be nice to do Jellyfin over https but not really required. Enter N8N, it also works on non standard ports and does have a SSL engine that should work with Cloudflare but it does not appear to be as easy to setup as first thought and any outside access will certainly need to be secured. Update Lychee has been destroyed to make way for a new Linux desktop

Certbot Management Cheat Sheet

If you need to check on the health of your "idling" certificates, use these commands on Raisin:

• Check Expiry Dates

sudo certbot certificates

• Test the Renewal Process

sudo certbot renew --dry-run

• Force an Immediate Renewal

sudo certbot renew --force-renewal

• View Scheduled Timer

systemctl list-timers *certbot*

• Check Logs for Failures

sudo tail -f /var/log/letsencrypt/letsencrypt.log

Acquire the Wildcard SSL Certificate and key from Letsencrypt

Initial features to be recognised

The requirement is to have one key and certificate pair for the entire *.seaoffate.net and seaoffate.net DNS name. First it should be noted that the wildcard SSL cert will only cover one subdomain deep IE it will be ok for wiki.seaoffate.net or files.seaoffate.net but it will not be any good for any deeper like other.wiki.seaoffate.net. If any deeper nesting of DNS subdomain names is required they will have to be on a separate request. It is not likely to be much of a problem for seaoffate.net but in a business setting it could be an issue.

The next thing to be considered is that the certs will only be valid for a set period of time then they will expire, they would still do the encryption but would return errors to the browser due to the expiry date, the solution is to have a bot retrieve new certificates before the old cert expires and indeed there are Certbots especially setup for this purpose.

Another feature that will have to be addressed is that Letsencrypt will need to verify that seaoffate is really being hosted at the correct address. If it was just a one off cert then there is a process where the request is made to Letsencrypt through a Certbot and Letsencrypt return a string that needs to be added to the DNS provider, in this case Cloudflare, and as the text file is published along with the rest of the DNS information Letsencrypt can be sure that the user is indeed the owner of the domain (several text strings may need to be published for verification). This manual process is not too difficult or even time consuming but it is a manual process so it would be better if it can be automated. Fortunately and unsurprisingly, Cloudflare have a solution to the automating domain ownership problem in the form of API keys. Obviously it would be better if the API keys were kept private as any leak would allow a hostile actor to impersonate my websites forever.

This will be a wildcard certificate so it can be used by anything.seaoffate.net and as many times as required. It could be copied to all of the webservers and used in place of the existing Cloudflare origin cert (cloudflare full strict only requires that it be a cert that cloudflare can recognise and as this will be a public cert it will work), this would make the warnings when using webservers directly disappear but it is not certain that this is even desired let alone required.

Install and setup Certbot

Raisin Installation

We need to be logged in to Raisin for this part.

First of all if there is a second attempt to install certbot on Raisin or if it is just changing things around it would be better to remove any old installation. Then install a fresh version again

sudo apt update
sudo apt remove --purge certbot python3-certbot-nginx 2>/dev/null || true # Remove old Certbot
sudo rm -rf /etc/letsencrypt/ # Clean up old certs and configs
sudo apt install -y snapd
sudo snap install --classic certbot
sudo snap install certbot-dns-cloudflare
sudo ln -s /snap/bin/certbot /usr/bin/certbot # Ensure certbot command is available

We should create dedicated system user for deployment (certbot-deploy)

sudo useradd --no-create-home --shell /bin/false certbot-deploy

and generate SSh key for certbot-deploy and as the user has no home dir Keys will be stored in a secure location under /etc/letsencrypt/:

sudo mkdir -p /etc/letsencrypt/certbot-deploy/.ssh/
sudo chown certbot-deploy:certbot-deploy /etc/letsencrypt/certbot-deploy/.ssh/
sudo chmod 700 /etc/letsencrypt/certbot-deploy/.ssh/
sudo -u certbot-deploy ssh-keygen -t rsa -b 4096 -f /etc/letsencrypt/certbot-deploy/.ssh/id_rsa -N ""

Grant certbot-deploy sudo Access for ssh/scp on raisin to allow certbot-deploy to use ssh and scp commands without a password.

sudo visudo -f /etc/sudoers.d/certbot-deploy

Add the following content to the sudoers file

# Allow certbot-deploy user to execute ssh and scp without password
certbot-deploy ALL=(ALL) NOPASSWD: /usr/bin/ssh
certbot-deploy ALL=(ALL) NOPASSWD: /usr/bin/scp

Save and exit. make sure that the system allows a clean save and does not give options to edit or save indicating there is a syntax error in the commands.

Lychee Installation

Update Lychee has been destroyed in favour of a new Linux desktop

We need to be logged in to Lychee for this part

Add nigel user to ssl-cert group (if not already): (To ensure nigel can read the letsencrypt_central directory)

sudo usermod -aG ssl-cert nigel

We need to Log out and log back in to Lychee for group changes to take effect. Next we create helper scripts for sudoers (chown -R and chmod -R) These scripts bypass sudoers' sensitivity to recursive commands. The two scripts are chown_certs_dir.sh and chmod_certs_dir.sh

sudo nano /usr/local/sbin/chown_certs_dir.sh

with the contents

 #!/bin/bash ; 
 /usr/bin/chown -R nigel:nigel "$1"

save and exit and set permissions and ownership

sudo chmod 700 /usr/local/sbin/chown_certs_dir.sh
sudo chown root:root /usr/local/sbin/chown_certs_dir.sh

and the second script is created with

sudo nano /usr/local/sbin/chmod_certs_dir.sh

with the contents

 #!/bin/bash ; 
 /usr/bin/chmod -R 700 "$1"

save and exit and set permissions and ownership

sudo chmod 700 /usr/local/sbin/chmod_certs_dir.sh
sudo chown root:root /usr/local/sbin/chmod_certs_dir.sh

We will need to configure nigel's sudoers on Lychee for NOPASSWD access

sudo visudo -f /etc/sudoers.d/nigel_cert_push

with the following contents

 # Allow nigel to run specific commands for cert push via SSH without password
 # For directory creation
 nigel ALL=(ALL) NOPASSWD: /bin/mkdir -p /home/nigel/npm/letsencrypt/live/seaoffate.net

 # For chown -R and chmod -R using helper scripts
 nigel ALL=(ALL) NOPASSWD: /usr/local/sbin/chown_certs_dir.sh
 nigel ALL=(ALL) NOPASSWD: /usr/local/sbin/chmod_certs_dir.sh

 # For chmod 644 for fullchain.pem
 nigel ALL=(ALL) NOPASSWD: /bin/bash -c "/bin/chmod 644 /home/nigel/npm/letsencrypt/live/seaoffate.net/fullchain.pem"

 # For chmod 600 for privkey.pem
 nigel ALL=(ALL) NOPASSWD: /bin/bash -c "/bin/chmod 600 /home/nigel/npm/letsencrypt/live/seaoffate.net/privkey.pem"

 # For Nginx reload
 nigel ALL=(ALL) NOPASSWD: /usr/bin/systemctl reload nginx
 nigel ALL=(ALL) NOPASSWD: /usr/sbin/nginx -s reload

 # For scp (client-side for receiving files)
 nigel ALL=(ALL) NOPASSWD: /usr/bin/scp

Save and exit. Note: if the file does not save cleanly the syntax errors will need to be fixed.

Obtain Let's Encrypt Wildcard Certificates (on Raisin)

We need to be logged on to Raisin again for this part

We need to login to the Cloudflare control panel and go to the Profile section (from the username) then select "API Tokens" from the menu on the left. In the API Tokens main panel there should be a button "create Token" and when clicked leads to a screen with several API Token types that can be created. Of all those types none quite match so we "Create Custom Token" at the bottom.

This Custom token needs

• Name = Certbot_Renewal_Token_Net
• Permission = Zone - DNS - Edit
• Zone Resources = Include = Specific Zone - seaoffate.net

leave the other fields at default or blank. Click Continue to summary then Create Token The next screen will show the Token and a example curn command to demonstrate how it is used. Make sure that the key is copied to keepass as it will not be able to see it again once this page is closed. We will need to create another token for seaoffate.uk

Back on Raisin we need a secure directory with files for each Token

sudo mkdir -p /etc/letsencrypt/cloudflare && sudo chmod 700 /etc/letsencrypt/cloudflare
sudo nano /etc/letsencrypt/cloudflare/cloudflare_net.ini

with the contents

dns_cloudflare_api_token = YOUR_CLOUDFLARE_API_TOKEN_NET

then create a similar file for the .uk token

sudo nano /etc/letsencrypt/cloudflare/cloudflare_uk.ini

with the contents

dns_cloudflare_api_token = YOUR_CLOUDFLARE_API_TOKEN_UK

Set permissions:

sudo chmod 600 /etc/letsencrypt/cloudflare/cloudflare_net.ini && sudo chmod 600 /etc/letsencrypt/cloudflare/cloudflare_uk.ini

Obtain Wildcard Certificates and also set up the automatic renewal in the background.

sudo certbot certonly --dns-cloudflare --dns-cloudflare-credentials /etc/letsencrypt/cloudflare/cloudflare_net.ini -d seaoffate.net -d *.seaoffate.net --agree-tos -n -m [email protected]
sudo certbot certonly --dns-cloudflare --dns-cloudflare-credentials /etc/letsencrypt/cloudflare/cloudflare_uk.ini -d seaoffate.uk -d *.seaoffate.uk --agree-tos -n -m [email protected]

We can confirm the timers are active with the command

systemctl list-timers | grep certbot

Still on Raisin we create a script to push the certs to Lychee

sudo nano /etc/letsencrypt/renewal-hooks/deploy/push_certs_to_lychee.sh

with the contents

 #!/bin/bash
 set -e # Exit immediately if a command exits with a non-zero status
 set -x # Enable command tracing for debugging (can remove later)

 # Configuration variables
 SSH_USER="nigel" # Your username on Lychee (for SSH login)
 REMOTE_HOST="192.168.100.27" # Lychee's IP address
 REMOTE_CERT_PATH="/home/$SSH_USER/npm/letsencrypt/live/seaoffate.net" # Correct path on Lychee

 # Certbot provides these environment variables during deploy hooks
 # RENEWED_LINEAGE: Path to /etc/letsencrypt/live/seaoffate.net for the *renewed* cert
 # RENEWED_DOMAINS: Space-separated list of domains that were renewed

 log_file="/var/log/certbot-deploy-hook/certbot-deploy-hook.log"

 # Ensure log file exists and is writable by root (this script runs as root)
 touch "$log_file"
 chmod 644 "$log_file"

 echo "$(date '+%Y-%m-%d %H:%M:%S') - Starting deploy hook for $RENEWED_DOMAINS" >> "$log_file"

 # --- Remote Commands on Lychee (executed via SSH as SSH_USER, using sudo for root actions) ---
 # Call helper scripts to prepare the directory and set its ownership
 ssh "$SSH_USER@$REMOTE_HOST" << EOF >> "$log_file" 2>&1
 sudo /bin/mkdir -p "$REMOTE_CERT_PATH"
 sudo /usr/local/sbin/chown_certs_dir.sh "$REMOTE_CERT_PATH"
 sudo /usr/local/sbin/chmod_certs_dir.sh "$REMOTE_CERT_PATH"
 EOF

 if [ $? -ne 0 ]; then
     echo "ERROR: Remote directory prep failed for $REMOTE_CERT_PATH on $REMOTE_HOST" >> "$log_file"
     exit 1
 fi

 # --- Local SCP Commands (executed by 'root' on Raisin) ---
 # Transfer files from Raisin to Lychee
 # This requires 'root' on Raisin to have passwordless SSH to 'nigel@REMOTE_HOST'.
 scp "$RENEWED_LINEAGE/fullchain.pem" "$SSH_USER@$REMOTE_HOST:$REMOTE_CERT_PATH/"
 scp "$RENEWED_LINEAGE/privkey.pem" "$SSH_USER@$REMOTE_HOST:$REMOTE_CERT_PATH/"

 if [ $? -ne 0 ]; then
     echo "ERROR: Failed to copy certificate files from $RENEWED_LINEAGE to $REMOTE_HOST" >> "$log_file"
     exit 1
 fi

 # --- Remote Commands on Lychee (set permissions on copied files and reload Nginx) ---
 # Files are copied as 'nigel', so 'nigel' owns them. Now 'nigel' sets perms and reloads Nginx.
 ssh "$SSH_USER@$REMOTE_HOST" << EOF >> "$log_file" 2>&1
 /bin/chmod 644 "$REMOTE_CERT_PATH/fullchain.pem"
 /bin/chmod 600 "$REMOTE_CERT_PATH/privkey.pem"
 sudo /usr/bin/systemctl reload nginx # Needs NOPASSWD sudo for nigel on Lychee
 EOF

 if [ $? -ne 0 ]; then
     echo "ERROR: Remote permissions or Nginx reload failed on $REMOTE_HOST" >> "$log_file"
     exit 1
 fi

 echo "$(date '+%Y-%m-%d %H:%M:%S') - Deploy hook completed successfully for $RENEWED_DOMAINS" >> "$log_file"
 exit 0

To test the deploy we need to clear the old log file:

sudo rm -f /var/log/certbot-deploy-hook/certbot-deploy-hook.log

Then execute the deploy-hook script but we also need to manually set RENEWED_LINEAGE and RENEWED_DOMAINS for this test run

sudo RENEWED_LINEAGE="/etc/letsencrypt/live/seaoffate.net" RENEWED_DOMAINS="seaoffate.net *.seaoffate.net" /etc/letsencrypt/renewal-hooks/deploy/push_certs_to_lychee.sh

This command must now run cleanly without any password prompts or permission errors in the log. Check the log file on raisin for success:

sudo cat /var/log/certbot-deploy-hook/certbot-deploy-hook.log

Look for "Starting deploy hook," and "Deploy hook completed successfully." There should be no "ERROR" messages from ssh, scp, or sudo calls within the log. On Lychee

ls -l /home/nigel/npm/letsencrypt/live/seaoffate.net/ # Check if fullchain.pem and privkey.pem are there
sudo cat /home/nigel/npm/letsencrypt/live/seaoffate.net/fullchain.pem | awk 'BEGIN {c=0;} /BEGIN CERT/{c++} END {print c}' # Check if it's a full chain (should be 2 or more)
sudo systemctl status nginx # Check if Nginx reloaded recently (look at timestamps in logs)

The key and cert should now be echoed to the screen.

With the display of the cert and key the auto renewal and push to Lychee is complete

Updated Information after Lychee VM removal

Lychee has been removed as a VM so all of the setup for Letsencrypt for Lychee has been purged.

Current "Ready" State

As of February 2026, the Certbot installation on Raisin is fully configured but currently "idling" because Nginx is using Cloudflare Origin Certificates.

• The Engine: Certbot is installed via snap (the 2026 standard) and is linked to the Cloudflare DNS API.
• The Credentials: Secure API tokens for both seaoffate.net and seaoffate.uk are stored in /etc/letsencrypt/cloudflare/.
• The Certificates: Valid wildcard certificates for both domains are currently stored in /etc/letsencrypt/live/. Even if Nginx isn't using them, Certbot is successfully renewing them in the background.

The Renewal Logic (The "Hooks")

The system is now "Clean."

• Success: A dry-run confirms that the renewal process works perfectly without trying to talk to any external VMs (like the old Lychee).
• Trigger: When a renewal happens, Raisin only performs one action: sudo systemctl reload nginx. This is set in the /etc/letsencrypt/renewal/*.conf files.

How to Switch from Cloudflare to Let's Encrypt

If we ever need to use the Let's Encrypt certs instead of the Cloudflare Origin ones (e.g., for a service not behind the proxy), follow these three steps:

• Update the Nginx Config
  • Change the certificate paths in your Nginx site configuration in /etc/nginx/sites-available/somewebsite.conf

# FROM (Cloudflare Origin):
ssl_certificate /etc/ssl/certs/seaoffatenet.crt;
ssl_certificate_key /etc/ssl/private/seaoffatenet.key;

# TO (Let's Encrypt Wildcard):
ssl_certificate /etc/letsencrypt/live/seaoffate.net/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/seaoffate.net/privkey.pem;

• Ensure Permission Access
  • Let's Encrypt directories are strictly locked. Ensure Nginx can read them

sudo chown -R root:www-data /etc/letsencrypt/live/
sudo chown -R root:www-data /etc/letsencrypt/archive/

• Verify and Reload

sudo nginx -t && sudo systemctl reload nginx

Summary of "Inactive" Features

• Remote Push: Deleted. Raisin no longer attempts to scp certificates to other IPs
• User Accounts: The certbot-deploy user has been deleted. Any future remote pushing will require creating a new SSH keypair and user.

Conclusion

To restart using Letsencrypt we would have to create and configure the script to push the new SSL certs to the new VM but as the certbot will is currently working we will not need to set that up again