Letsencrypt Wildcard SSL Cert

From Sea of Fate
Jump to navigationJump to search

Introduction

The time has come when we need to have SSL certs that can be used unrepentantly of Cloudflare. Cloudflare's proxy for webservers does work well but on the free tier they only listen to 443 and anything else will be rejected and return an invalid response. For some services like VPN (either OpenVPN or Wireguard) it does not matter as they will do their own encryption. However, services like Jellyfin use alternate ports for both http and https traffic so cannot be proxied by Cloudflare, to date it has not really mattered because streaming to remote locations is not really required too much and if it is we can use a VPN link if privacy is an issue. It would be nice to do Jellyfin over https but not really required. Enter N8N, it also works on non standard ports and does have a SSL engine that should work with Cloudflare but it does not appear to be as easy to setup as first thought and any outside access will certainly need to be secured.

Acquire the Wildcard SSL Certificate and key from Letsencrypt

Initial features to be recognised

The requirement is to have one key and certificate pair for the entire *.seaoffate.net and seaoffate.net DNS name. First it should be noted that the wildcard SSL cert will only cover one subdomain deep IE it will be ok for wiki.seaoffate.net or files.seaoffate.net but it will not be any good for any deeper like other.wiki.seaoffate.net. If any deeper nesting of DNS subdomain names is required they will have to be on a separate request. It is not likely to be much of a problem for seaoffate.net but in a business setting it could be an issue.

The next thing to be considered is that the certs will only be valid for a set period of time then they will expire, they would still do the encryption but would return errors to the browser due to the expiry date, the solution is to have a bot retrieve new certificates before the old cert expires and indeed there are Certbots especially setup for this purpose.

Another feature that will have to be addressed is that Letsencrypt will need to verify that seaoffate is really being hosted at the correct address. If it was just a one off cert then there is a process where the request is made to Letsencrypt through a Certbot and Letsencrypt return a string that needs to be added to the DNS provider, in this case Cloudflare, and as the text file is published along with the rest of the DNS information Letsencrypt can be sure that the user is indeed the owner of the domain (several text strings may need to be published for verification). This manual process is not too difficult or even time consuming but it is a manual process so it would be better if it can be automated. Fortunately and unsurprisingly, Cloudflare have a solution to the automating domain ownership problem in the form of API keys. Obviously it would be better if the API keys were kept private as any leak would allow a hostile actor to impersonate my websites forever.

This will be a wildcard certificate so it can be used by anything.seaoffate.net and as many times as required. It could be copied to all of the webservers and used in place of the existing Cloudflare origin cert (cloudflare full strict only requires that it be a cert that cloudflare can recognise and as this will be a public cert it will work), this would make the warnings when using webservers directly disappear but it is not certain that this is even desired let alone required.

Install and setup Certbot

Raisin Installation

We need to be logged in to Raisin for this part.

First of all if there is a second attempt to install certbot on Raisin or if it is just changing things around it would be better to remove any old installation. Then install a fresh version again 

sudo apt update
sudo apt remove --purge certbot python3-certbot-nginx 2>/dev/null || true # Remove old Certbot
sudo rm -rf /etc/letsencrypt/ # Clean up old certs and configs
sudo apt install -y snapd
sudo snap install --classic certbot
sudo snap install certbot-dns-cloudflare
sudo ln -s /snap/bin/certbot /usr/bin/certbot # Ensure certbot command is available

We should create dedicated system user for deployment (certbot-deploy) 

sudo useradd --no-create-home --shell /bin/false certbot-deploy

and generate SSh key for certbot-deploy and as the user has no home dir Keys will be stored in a secure location under /etc/letsencrypt/: 

sudo mkdir -p /etc/letsencrypt/certbot-deploy/.ssh/
sudo chown certbot-deploy:certbot-deploy /etc/letsencrypt/certbot-deploy/.ssh/
sudo chmod 700 /etc/letsencrypt/certbot-deploy/.ssh/
sudo -u certbot-deploy ssh-keygen -t rsa -b 4096 -f /etc/letsencrypt/certbot-deploy/.ssh/id_rsa -N ""

Grant certbot-deploy sudo Access for ssh/scp on raisin to allow certbot-deploy to use ssh and scp commands without a password. 

sudo visudo -f /etc/sudoers.d/certbot-deploy

Add the following content to the sudoers file 

# Allow certbot-deploy user to execute ssh and scp without password
certbot-deploy ALL=(ALL) NOPASSWD: /usr/bin/ssh
certbot-deploy ALL=(ALL) NOPASSWD: /usr/bin/scp

Save and exit. make sure that the system allows a clean save and does not give options to edit or save indicating there is a syntax error in the commands.

Lychee Installation

We need to be logged in to Lychee for this part

Add nigel user to ssl-cert group (if not already): (To ensure nigel can read the letsencrypt_central directory) 

sudo usermod -aG ssl-cert nigel

We need to Log out and log back in to Lychee for group changes to take effect. Next we create helper scripts for sudoers (chown -R and chmod -R) These scripts bypass sudoers' sensitivity to recursive commands. The two scripts are chown_certs_dir.sh and chmod_certs_dir.sh 

sudo nano /usr/local/sbin/chown_certs_dir.sh

with the contents 

 #!/bin/bash ; 
 /usr/bin/chown -R nigel:nigel "$1"

save and exit and set permissions and ownership 

sudo chmod 700 /usr/local/sbin/chown_certs_dir.sh
sudo chown root:root /usr/local/sbin/chown_certs_dir.sh

and the second script is created with 

sudo nano /usr/local/sbin/chmod_certs_dir.sh

with the contents 

 #!/bin/bash ; 
 /usr/bin/chmod -R 700 "$1"

save and exit and set permissions and ownership 

sudo chmod 700 /usr/local/sbin/chmod_certs_dir.sh
sudo chown root:root /usr/local/sbin/chmod_certs_dir.sh

We will need to configure nigel's sudoers on Lychee for NOPASSWD access 

sudo visudo -f /etc/sudoers.d/nigel_cert_push

with the following contents 

 # Allow nigel to run specific commands for cert push via SSH without password
 # For directory creation
 nigel ALL=(ALL) NOPASSWD: /bin/mkdir -p /home/nigel/npm/letsencrypt/live/seaoffate.net

 # For chown -R and chmod -R using helper scripts
 nigel ALL=(ALL) NOPASSWD: /usr/local/sbin/chown_certs_dir.sh
 nigel ALL=(ALL) NOPASSWD: /usr/local/sbin/chmod_certs_dir.sh

 # For chmod 644 for fullchain.pem
 nigel ALL=(ALL) NOPASSWD: /bin/bash -c "/bin/chmod 644 /home/nigel/npm/letsencrypt/live/seaoffate.net/fullchain.pem"

 # For chmod 600 for privkey.pem
 nigel ALL=(ALL) NOPASSWD: /bin/bash -c "/bin/chmod 600 /home/nigel/npm/letsencrypt/live/seaoffate.net/privkey.pem"

 # For Nginx reload
 nigel ALL=(ALL) NOPASSWD: /usr/bin/systemctl reload nginx
 nigel ALL=(ALL) NOPASSWD: /usr/sbin/nginx -s reload

 # For scp (client-side for receiving files)
 nigel ALL=(ALL) NOPASSWD: /usr/bin/scp

Save and exit. Note: if the file does not save cleanly the syntax errors will need to be fixed.

Obtain Let's Encrypt Wildcard Certificates (on Raisin)

We need to be logged on to Raisin again for this part

We need to login to the Cloudflare control panel and go to the Profile section (from the username) then select "API Tokens" from the menu on the left. In the API Tokens main panel there should be a button "create Token" and when clicked leads to a screen with several API Token types that can be created. Of all those types none quite match so we "Create Custom Token" at the bottom.

This Custom token needs

  • Name = Certbot_Renewal_Token_Net
  • Permission = Zone - DNS - Edit
  • Zone Resources = Include = Specific Zone - seaoffate.net

leave the other fields at default or blank. Click Continue to summary then Create Token The next screen will show the Token and a example curn command to demonstrate how it is used. Make sure that the key is copied to keepass as it will not be able to see it again once this page is closed. We will need to create another token for seaoffate.uk

Back on Raisin we need a secure directory with files for each Token 

sudo mkdir -p /etc/letsencrypt/cloudflare && sudo chmod 700 /etc/letsencrypt/cloudflare
sudo nano /etc/letsencrypt/cloudflare/cloudflare_net.ini

with the contents 

dns_cloudflare_api_token = YOUR_CLOUDFLARE_API_TOKEN_NET

then create a similar file for the .uk token 

sudo nano /etc/letsencrypt/cloudflare/cloudflare_uk.ini

with the contents 

dns_cloudflare_api_token = YOUR_CLOUDFLARE_API_TOKEN_UK

Set permissions: 

sudo chmod 600 /etc/letsencrypt/cloudflare/cloudflare_net.ini && sudo chmod 600 /etc/letsencrypt/cloudflare/cloudflare_uk.ini

Obtain Wildcard Certificates and also set up the automatic renewal in the background. 

sudo certbot certonly --dns-cloudflare --dns-cloudflare-credentials /etc/letsencrypt/cloudflare/cloudflare_net.ini -d seaoffate.net -d *.seaoffate.net --agree-tos -n -m [email protected]
sudo certbot certonly --dns-cloudflare --dns-cloudflare-credentials /etc/letsencrypt/cloudflare/cloudflare_uk.ini -d seaoffate.uk -d *.seaoffate.uk --agree-tos -n -m [email protected]

We can confirm the timers are active with the command 

systemctl list-timers | grep certbot

Still on Raisin we create a script to push the certs to Lychee 

sudo nano /etc/letsencrypt/renewal-hooks/deploy/push_certs_to_lychee.sh

with the contents 

 #!/bin/bash
 set -e # Exit immediately if a command exits with a non-zero status
 set -x # Enable command tracing for debugging (can remove later)

 # Configuration variables
 SSH_USER="nigel" # Your username on Lychee (for SSH login)
 REMOTE_HOST="192.168.100.27" # Lychee's IP address
 REMOTE_CERT_PATH="/home/$SSH_USER/npm/letsencrypt/live/seaoffate.net" # Correct path on Lychee

 # Certbot provides these environment variables during deploy hooks
 # RENEWED_LINEAGE: Path to /etc/letsencrypt/live/seaoffate.net for the *renewed* cert
 # RENEWED_DOMAINS: Space-separated list of domains that were renewed

 log_file="/var/log/certbot-deploy-hook/certbot-deploy-hook.log"

 # Ensure log file exists and is writable by root (this script runs as root)
 touch "$log_file"
 chmod 644 "$log_file"

 echo "$(date '+%Y-%m-%d %H:%M:%S') - Starting deploy hook for $RENEWED_DOMAINS" >> "$log_file"

 # --- Remote Commands on Lychee (executed via SSH as SSH_USER, using sudo for root actions) ---
 # Call helper scripts to prepare the directory and set its ownership
 ssh "$SSH_USER@$REMOTE_HOST" << EOF >> "$log_file" 2>&1
 sudo /bin/mkdir -p "$REMOTE_CERT_PATH"
 sudo /usr/local/sbin/chown_certs_dir.sh "$REMOTE_CERT_PATH"
 sudo /usr/local/sbin/chmod_certs_dir.sh "$REMOTE_CERT_PATH"
 EOF

 if [ $? -ne 0 ]; then
     echo "ERROR: Remote directory prep failed for $REMOTE_CERT_PATH on $REMOTE_HOST" >> "$log_file"
     exit 1
 fi

 # --- Local SCP Commands (executed by 'root' on Raisin) ---
 # Transfer files from Raisin to Lychee
 # This requires 'root' on Raisin to have passwordless SSH to 'nigel@REMOTE_HOST'.
 scp "$RENEWED_LINEAGE/fullchain.pem" "$SSH_USER@$REMOTE_HOST:$REMOTE_CERT_PATH/"
 scp "$RENEWED_LINEAGE/privkey.pem" "$SSH_USER@$REMOTE_HOST:$REMOTE_CERT_PATH/"

 if [ $? -ne 0 ]; then
     echo "ERROR: Failed to copy certificate files from $RENEWED_LINEAGE to $REMOTE_HOST" >> "$log_file"
     exit 1
 fi

 # --- Remote Commands on Lychee (set permissions on copied files and reload Nginx) ---
 # Files are copied as 'nigel', so 'nigel' owns them. Now 'nigel' sets perms and reloads Nginx.
 ssh "$SSH_USER@$REMOTE_HOST" << EOF >> "$log_file" 2>&1
 /bin/chmod 644 "$REMOTE_CERT_PATH/fullchain.pem"
 /bin/chmod 600 "$REMOTE_CERT_PATH/privkey.pem"
 sudo /usr/bin/systemctl reload nginx # Needs NOPASSWD sudo for nigel on Lychee
 EOF

 if [ $? -ne 0 ]; then
     echo "ERROR: Remote permissions or Nginx reload failed on $REMOTE_HOST" >> "$log_file"
     exit 1
 fi

 echo "$(date '+%Y-%m-%d %H:%M:%S') - Deploy hook completed successfully for $RENEWED_DOMAINS" >> "$log_file"
 exit 0

To test the deploy we need to clear the old log file: 

sudo rm -f /var/log/certbot-deploy-hook/certbot-deploy-hook.log

Then execute the deploy-hook script but we also need to manually set RENEWED_LINEAGE and RENEWED_DOMAINS for this test run 

sudo RENEWED_LINEAGE="/etc/letsencrypt/live/seaoffate.net" RENEWED_DOMAINS="seaoffate.net *.seaoffate.net" /etc/letsencrypt/renewal-hooks/deploy/push_certs_to_lychee.sh

This command must now run cleanly without any password prompts or permission errors in the log. Check the log file on raisin for success: 

sudo cat /var/log/certbot-deploy-hook/certbot-deploy-hook.log

Look for "Starting deploy hook," and "Deploy hook completed successfully." There should be no "ERROR" messages from ssh, scp, or sudo calls within the log. On Lychee 

ls -l /home/nigel/npm/letsencrypt/live/seaoffate.net/ # Check if fullchain.pem and privkey.pem are there
sudo cat /home/nigel/npm/letsencrypt/live/seaoffate.net/fullchain.pem | awk 'BEGIN {c=0;} /BEGIN CERT/{c++} END {print c}' # Check if it's a full chain (should be 2 or more)
sudo systemctl status nginx # Check if Nginx reloaded recently (look at timestamps in logs)

The key and cert should now be echoed to the screen

Retrieved from "https://wiki.seaoffate.net/index.php?title=Letsencrypt_Wildcard_SSL_Cert&oldid=559"