Unified Monitoring Stack: Difference between revisions

From Sea of Fate
Jump to navigationJump to search
← Older edit
 
(7 intermediate revisions by the same user not shown)
Line 469: Line 469:
As can be noted in the YAML file there are two lines that look suspicious the new accounts and the admin token. These are left as they are for the initial setup but we will change them now. first the new accounts needs to be remedied so that if someone on the Internet stumbles across our vault they don't start filling our password manager with bogus junk. Obviously, we need to create our own account so we logon to <nowiki>https://vault.seaoffate.net</nowiki>. At the vault warden logon screen create the account/s as required. Once all of the accounts are created we can change the line in the environment "SIGNUPS_ALLOWED=true" to SIGNUPS_ALLOWED=false and restart the application. From this point on no new accounts can be created, while the logon screen does continue to show the option to do so it will not actually complete the new account creation process.
As can be noted in the YAML file there are two lines that look suspicious the new accounts and the admin token. These are left as they are for the initial setup but we will change them now. first the new accounts needs to be remedied so that if someone on the Internet stumbles across our vault they don't start filling our password manager with bogus junk. Obviously, we need to create our own account so we logon to <nowiki>https://vault.seaoffate.net</nowiki>. At the vault warden logon screen create the account/s as required. Once all of the accounts are created we can change the line in the environment "SIGNUPS_ALLOWED=true" to SIGNUPS_ALLOWED=false and restart the application. From this point on no new accounts can be created, while the logon screen does continue to show the option to do so it will not actually complete the new account creation process.


====Securing the Admin Logon===
====Securing the Admin Logon====


The next thing to address is the "ADMIN_TOKEN=yoursecret_change_this # Used for the /admin panel" line. The admin logon is accessed from the URL <nowiki>https://vault.seaoffate.net/admin</nowiki> and is be used to configure options for the Vault warden users. There is no username just a password. To avoid posting a password on the Dockge screen we use a password encryption routine supplied by the Vault warden application. to create a password follow these steps
The next thing to address is the "ADMIN_TOKEN=yoursecret_change_this # Used for the /admin panel" line. The admin logon is accessed from the URL <nowiki>https://vault.seaoffate.net/admin</nowiki> and is be used to configure options for the Vault warden users. There is no username just a password. To avoid posting a password on the Dockge screen we use a password encryption routine supplied by the Vault warden application. to create a password follow these steps
Line 496: Line 496:
* From the terminal of Raisin we need a passwordless logon to mango for the backup script to run
* From the terminal of Raisin we need a passwordless logon to mango for the backup script to run
  ssh-copy-id [email protected]
  ssh-copy-id [email protected]
** If an error like "/usr/bin/ssh-copy-id: ERROR: No identities found" use the keygen routine
* If an error like "/usr/bin/ssh-copy-id: ERROR: No identities found" use the keygen routine (just accept the defaults)
  ssh-keygen -t ed25519 -C "nigel@tayberry"
  ssh-keygen -t ed25519 -C "nigel@raisin"
* Create the backup script
* Create the backup script and file locations
  sudo nano ~/scripts/backup-vault.sh
mkdir -p ~/scripts
mkdir -p ~/logs
  nano ~/scripts/backup-vault.sh
* Copy the script to the file
* Copy the script to the file
  #!/bin/bash
  #!/bin/bash
Line 505: Line 507:
  # Configuration
  # Configuration
  REMOTE_USER="nigel"
  REMOTE_USER="nigel"
  REMOTE_HOST="mango.seaoffate.local" # Mango
  REMOTE_HOST="mango.seaoffate.local"
  REMOTE_DIR="/opt/dockge/stacks/vaultwarden/data"
  REMOTE_DIR="/opt/dockge/stacks/vaultwarden/data"
  BACKUP_DIR="/home/nigel/backups/vaultwarden"
  BACKUP_DIR="/home/nigel/backups/vaultwarden"
Line 515: Line 517:
  echo "Starting Vaultwarden backup from Mango..."
  echo "Starting Vaultwarden backup from Mango..."
  #
  #
  # Use rsync to pull the data folder.
  # Sync files
# --delete ensures if you delete a file on Mango, it eventually clears on Raisin
# We exclude the 'attachments' folder if it's too big, but usually, it's fine.
  if rsync -avz -e ssh "$REMOTE_USER@$REMOTE_HOST:$REMOTE_DIR" "$BACKUP_DIR/latest/"; then
  if rsync -avz -e ssh "$REMOTE_USER@$REMOTE_HOST:$REMOTE_DIR" "$BACKUP_DIR/latest/"; then
     echo "Files synced successfully."
     echo "Files synced successfully."
    #
    # Create the timestamped archive
     tar -czf "$BACKUP_DIR/vault_backup_$TIMESTAMP.tar.gz" -C "$BACKUP_DIR/latest" .
     tar -czf "$BACKUP_DIR/vault_backup_$TIMESTAMP.tar.gz" -C "$BACKUP_DIR/latest" .
    #
    # Delete files older than 30 days
    find "$BACKUP_DIR" -name "vault_backup_*.tar.gz" -mtime +30 -delete
    #
     echo "Backup complete: $BACKUP_DIR/vault_backup_$TIMESTAMP.tar.gz"
     echo "Backup complete: $BACKUP_DIR/vault_backup_$TIMESTAMP.tar.gz"
  else
  else
Line 536: Line 542:
  sudo apt update && sudo apt install rsync -y
  sudo apt update && sudo apt install rsync -y
  exit
  exit
* When back on raisin we need to add a new crontab to run the script every night
* When back on raisin we need to add a new crontab (choose option 1 if asked) 
  crontab -e
  crontab -e
* Add the line to the bottom of  the list
* Add the line to the bottom of  the list to run the script every morning at 03:00
  # Every night at 3:00 AM, pull the vault data from Mango
  # Every morning at 3:00 AM, pull the vault data from Mango
  0 3 * * * /home/nigel/scripts/backup-vault.sh >> /home/nigel/logs/vault-backup.log 2>&1
  0 3 * * * /home/nigel/scripts/backup-vault.sh >> /home/nigel/logs/vault-backup.log 2>&1
* save & close, we can test the script with
* save & close, we can test the script with

Latest revision as of 00:26, 25 February 2026

📖Introduction

Mango, located at 192.168.110.133 on the Infra network, is the unified successor to the Prometheus & Grafana and Victoria triad. It serves as the central hub for the Home Lab's observability. Mango natively scrapes metrics from all Virtual Machines, the Proxmox host(Pear) and the services, stores them in a high-performance VictoriaMetrics time-series database, and provides a Grafana interface for visualization. As an additional step we can manage the Docker installations that have Dockge with one master Dockge installation on Mango and possibly manage any Minecraft servers from this same host.

By consolidating these services, we reduce network overhead and simplify the management of our monitoring infrastructure while maintaining 12-month data retention on a dedicated 500GB storage pool.

🚦Security & Network Architecture

Mango sits within the Infra network. Because it aggregates data from every host in the lab, it is a high-value target.

  • Web Interfaces: Grafana (Port 3000) and VictoriaMetrics VMUI (Port 8428) are restricted via pfSense to be accessible only from the MGT network (Cinnamon/Lemon).
  • Scraping Flow: Mango acts as the source for all scrape requests. pfSense rules must allow Mango to reach out to Production, VPN, and Terminal networks on specific exporter ports (9100, 9113, 9117, etc.).
  • Storage Pool: Data is stored on a dedicated 500GB virtual disk (PearPool), mounted at /mnt/metrics_data to ensure that metric growth never impacts the OS root partition.

🏛️Environment & Storage Setup

The VM was created using the Debian Gold Master template.

  • Hostname: Mango
  • IP/Gateway: 192.168.110.133 / 192.168.110.1
  • Disk 1 (OS): 32GB
  • Disk 2 (Data): 500GB (Added via Proxmox)

Storage Initialization To handle the long-term metrics, the 500GB disk was initialized and mounted: 

# Identify disk (sdb), format, and mount
sudo mkfs.ext4 /dev/sdb
sudo mkdir -p /mnt/metrics_data
sudo mount /dev/sdb /mnt/metrics_data
# Ensure persistence in /etc/fstab
/dev/sdb  /mnt/metrics_data  ext4  defaults  0  2

🔧Installation

⚡VictoriaMetrics Installation

VictoriaMetrics was installed as a native binary (not Docker) to replace both the Prometheus scraper and the Victoria storage VM.

  • User & Directory Setup
sudo useradd --no-create-home --shell /bin/false victoriametrics
sudo mkdir /etc/victoriametrics
sudo chown -R victoriametrics:victoriametrics /etc/victoriametrics /mnt/metrics_data
  • Binary Installation

Binaries were retrieved from the VictoriaMetrics GitHub. 

wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.xx.x/victoria-metrics-linux-amd64-v1.xx.x.tar.gz
tar -xvf victoria-metrics-linux-amd64-v1.xx.x.tar.gz
sudo mv victoria-metrics-prod /usr/local/bin/victoriametrics
sudo chown victoriametrics:victoriametrics /usr/local/bin/victoriametrics
  • Service Configuration
sudo nano /etc/systemd/system/victoriametrics.service

[Service]
ExecStart=/usr/local/bin/victoriametrics \
  --storageDataPath=/mnt/metrics_data \
  --retentionPeriod=12 \
  --promscrape.config=/etc/victoriametrics/prometheus.yml \
  --httpListenAddr=0.0.0.0:8428

Note: The --retentionPeriod=12 ensures one year of history.

🔍Scraping Configuration (prometheus.yml)

VictoriaMetrics uses the standard Prometheus YAML format for its scraper. The file was copied from the older Prometheus host Pineapple and copied to: 

sudo nano /etc/victoriametrics/prometheus.yml

Key Change: The evaluation_interval directive was removed as it is not natively supported by the VictoriaMetrics single-binary scraper (it expects vmalert for that).

🧪Target Jobs

The configuration includes the legacy fleet plus the new 2026 additions:

  • Infrastructure: Mango (Self), CTNS1.
  • Production:
    • Reverse proxy (Nginx) Raisin
    • Webservers (Apache) Plum, Satsuma, Fig
    • Database server (MySQL) Mandarin
  • New 2026 Hosts: Blackcurrant (Data & Archive), Quince (AI/Media), Tayberry (OpenAlex).
  • Gaming: Apple & Cherry (Minecraft Servers).
  • Terminals:
    • (NoMachine) Kiwiberry
    • (XRDP), Kapok
    • (Windows, RDP) Wahoo/Walnut .

Scrape Interval: Set to 120s to balance data resolution with disk I/O and longevity.

Adding the Dockge Targets

we had to update the /etc/victoriametrics/prometheus.yml to include the docker containers 

#scrape_configs:
  - job_name: 'docker_containers'
    static_configs:
      - targets:
          - 'quince.seaoffate.net:8080'      # cAdvisor (AI Stack)
          - 'blackberry.seaoffate.net:8080'  # cAdvisor (Data Archive)
          - 'tayberry.seaoffate.net:8080'    # cAdvisor (OpenAlex)

  - job_name: 'gpu_metrics'
    static_configs:
      - targets: ['quince.seaoffate.net:9400'] # DCGM Exporter

  - job_name: 'jellyfin'
    metrics_path: '/metrics' # Crucial: tells VM where to look on port 8096
    static_configs:
      - targets: ['quince.seaoffate.net:8096']

Target Agent Installation (Scrapers)

For Mango to collect data, each target VM must run a specific exporter. Most Linux hosts use the node_exporter for OS metrics, while application-specific exporters are used for Nginx, Apache, and MySQL.

Linux Node Exporter (Standard for all VMs)

Installed on all Linux hosts (Raisin, Plum, Satsuma, Apple, Cherry, etc.) to monitor CPU, RAM, and Disk. Any hosts that don't show on the targets webpage need to have the agent installed.

  • Install via APT
sudo apt update && sudo apt install -y prometheus-node-exporter
  • Enable and Start
sudo systemctl enable --now prometheus-node-exporter
  • Verification (Run on target VM)
curl http://localhost:9100/metrics
  • Firewall Requirement: Target VM must allow Inbound TCP 9100 from Mango (192.168.110.133).

Nginx (Raisin)

Used to monitor active connections and request rates. the Nginx exporter is a standalone binary that talks to Nginx's stub_status module.

  • Enable Nginx Status: On Raisin, edit the Nginx config (e.g., /etc/nginx/sites-available/default) and add this block:
server {
    listen 127.0.0.1:8080;
    location /metrics {
        stub_status on;
        access_log off;
        allow 127.0.0.1;
        deny all;
    }
}

sudo nginx -s reload
  • Install & Run Exporter
wget https://github.com/nginx/nginx-prometheus-exporter/releases/latest/download/nginx-prometheus-exporter_0.11.0_linux_amd64.tar.gz
tar -xvf nginx-prometheus-exporter_*.tar.gz
sudo mv nginx-prometheus-exporter /usr/local/bin/
  • Create a Systemd Service
sudo nano /etc/systemd/system/nginx-exporter.service

Paste this into the service file 

[Unit]
Description=Nginx Prometheus Exporter
After=network.target

[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/bin/nginx-prometheus-exporter \
    -nginx.scrape-uri=http://127.0.0.1:8080/metrics
Restart=always

[Install]
WantedBy=multi-user.target

Enable the service with 

sudo systemctl enable --now nginx-exporter

MySQL (Mandarin)

Used to monitor query throughput and database health.

  • Database User: Create a mysqld_exporter user in MySQL with PROCESS, REPLICATION CLIENT, SELECT privileges.
  • Configuration: Store credentials in /etc/.mysqld_exporter.cnf.
  • Service: Install prometheus-mysqld-exporter via APT.
  • Port: TCP 9104

Apache (Plum, Fig, Satsuma)

  • Enable Mod Status:
sudo a2enmod status.
  • Install Exporter:
sudo apt install prometheus-apache-exporter
  • Port: TCP 9117

Windows Exporter (Wahoo & Walnut)

For the Windows 11 desktops, we use the windows_exporter

  • Download: Latest .msi from the Prometheus Community GitHub.
  • Install: Run the installer; it defaults to port 9182.
  • Firewall: The installer typically adds a "Windows Firewall" exception automatically.

Docker & Container App Exporters

Since we are using Dockge to manage our containers on hosts like Quince (AI), Blackcurrant (Archive), and Tayberry (OpenAlex), we should standardize how metrics are pulled from these environments. The most efficient way to do this is to add cAdvisor to each of our Dockge stacks. This allows Mango to "see" inside the Docker engine of that specific VM and report on the health of every individual container (Ollama, Jellyfin, etc.).

Docker Container Monitoring (The Dockge Layer)

For every VM running Dockge, you need to add a Monitoring Stack or add these services to your existing stacks. cAdvisor is the primary agent here; it scrapes resource usage from the Docker socket.

  • Create a "Monitoring" Stack in Dockge for Blackberry and Tayberry: In the Dockge UI, create a new stack and use the following
    • Container_name: cadvisor
version: "3.8"
services:
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.47.0 # Use a version compatible with your kernel
    container_name: cadvisor
    restart: unless-stopped
    privileged: true
    ports:
      - 8080:8080
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
  • Create a "Monitoring" Stack in Dockge for Quince: In the Dockge UI, create a new stack and use the following
    • Container_name: cadvisor
services:
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.47.0
    container_name: cadvisor
    restart: unless-stopped
    privileged: true
    ports:
      - 8080:8080
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    devices:
      - /dev/kmsg
  nv-exporter:
    image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.5-3.4.0-ubuntu22.04
    container_name: nvidia_exporter
    restart: unless-stopped
    # Use 'command' to force it to listen on the network interface
    command:
      - -a
      - 0.0.0.0:9400
    ports:
      - 9400:9400 # Map host 9400 to container 9400
    cap_add:
      - SYS_ADMIN
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities:
                - gpu
networks: {}


Port Summary for Dockge Hosts (these ports must be opened in Pfsense for the exporters to report their status to Mango

  • 8080: cAdvisor (Container CPU/RAM/Network)
  • 9400: NVIDIA Exporter (GPU VRAM/Temp - Quince only)
Jellyfin monitoring

the switch to enable the metrics is not an option in the WebUI of our version of Jellyfin so we will have to enable it in the xml config. Since our Jellyfin config is mapped to /mnt/docker_data/jellyfin/config, finding the needle in the haystack is much easier. Because we are using the official Jellyfin image, the system.xml file is the principal config of the operation. On your host (Quince), the file you need to edit is right here: 

/mnt/docker_data/jellyfin/config/config/system.xml

We could stop the container and modify the .xml directly from the terminal on Quince. We can also use sed to find the false value and make it true without having to hunt through the XML manually, after stopping the container (within dockge) and making a backup: 

cd /mnt/docker_data/jellyfin/config/config/
cp system.xml system.xml.bak

Change EnableMetrics from false to true 

sed -i 's/<EnableMetrics>false<\/EnableMetrics>/<EnableMetrics>true<\/EnableMetrics>/' system.xml

Now the metrics are switched on we can restart the container app in dockge

Proxmox Host (Pear)

To monitor the physical hardware and ZFS pools:

  • Node Exporter: Installed directly on the Proxmox Debian host.
  • SMART Metrics: Use the smartctl_exporter_script.sh (as detailed in legacy notes) to pipe drive health into the node_exporter's textfile collector.

Post-Installation validation on Mango

After installing an agent on a target, confirm Mango sees it:

  • Open VMUI: http://mango:8428/targets.
  • Search for the hostname.
  • Status must be "UP". If "Connection Refused," check the service on the target; if "Timeout," check pfSense rules

📈 Grafana Installation

Grafana was installed on the same Mango host to provide the local visualization layer.

  • Repository & App Setup
sudo apt install -y apt-transport-https software-properties-common wget
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /usr/share/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/usr/share/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update && sudo apt install grafana -y
sudo systemctl enable --now grafana-server

🧩Network & Firewall Rules (pfSense)

To allow the new ports to function, pfSense was updated with a new Alias for Monitoring Ports:

  • 3000: Grafana UI (remains the same from the previous Grafana installation)
  • 8428: VictoriaMetrics UI/API ( The new port added for viewing of the scraping progress as was done by Prometheus web gui)
  • 9090: removed the older Prometheus webgui port

Critical Rules

  • MGT -> Mango: Allow ports 3000 & 8428 (Access from Cinnamon or other management console).
  • Mango -> All Networks:
    • Allow port 9100 (Node)
    • Allow port 9113 (Nginx)
    • Allow port 9117 (Apache)
    • Allow port 9104 (MySQL)
    • Allow port 9182 (Windows)
    • Allow port 8080 (Docker, cadvisor)
    • Allow port 9400 (NVIDIA DCGM)
    • Allow port 8096 (Jellyfin)

Obviously for clarity the above ports should all have aliases in Pfsense so the rules are easier to read.

🔦Verification Steps

  • Check Dockge: Ensure the cadvisor container shows as "Green/Running" in the Dockge UI.
  • Service Status: (Confirmed Active/Running).
sudo systemctl status victoriametrics

Targets Check: verify all hosts are Green/UP. We should see the new entries for ports 8080, 9400, etc for the docker containers 

http://mango:8428/targets

Data Source: In Grafana, added Prometheus data source pointing to 

http://localhost:8428.

Disk Write Check: confirms ingestion of samples to the PearPool disk. 

du -sh /mnt/metrics_data

We can verify the various agents are reporting with the curl command from mango. for example to test the docker container on Tayberry is working use: (docker app uses 8080 for its scraper) 

curl http://tayberry.seaoffate.net:8080/metrics

We should We could modify the example to test other agents by using a different hostname and/or a different port number for example Tayberry also uses the standard Linux Exporter on 9100 so we could have : 

curl http://tayberry.seaoffate.net:9100/metrics

and it should also present a wall of text from tayberry, assuming the the node exporter was installed. If any curl statements that don't work we should check the exporter is installed on the target, the appropriate Pfsense rule is setup and working, and that we have the correct port in the curl. If the curl works but the http://mango:8428/targets does not check the yaml file on mango

Summary of Legacy Retirement

With Mango fully operational:

  • Pineapple (.130) services stopped.
  • Granadilla (.131) services stopped.
  • Victoria (.132) services stopped.
  • Lychee identified as legacy and marked for rebuild via new Gold Master Template.

🐋Mango as a Dockge Master

We will also be installing Docker and Dockge so that we can manage all of the docker installations from a single master management interface rather than the three individual Dockge WebGUIs. It transforms Mango from just a monitoring server into a true Management Hub for the entire cluster. Since Mango is running a native VictoriaMetrics install, we need to be careful with Docker installation so it doesn't conflict with your existing systemd services.

The Strategy: The Manager's Desk

We aren't moving the actual containers (like Jellyfin or Ollama) to Mango—those stay on Quince, Blackberry, and Tayberry. Instead, we are setting up Dockge on Mango to act as the single interface that logs into the other three using the Dockge Agent or Remote Contexts.

Phase One Install Docker on Mango

Since Mango is on Debian, we’ll use the official Docker repository to ensure we get the latest Compose features that Dockge relies on. Because Trixie is a testing branch, your wiki needs to be very specific about the "Bookworm" override. Since Mango is running Debian Trixie (13 Testing), and Docker does not yet have a dedicated repository for this codename, we must manually point the repository to the stable Bookworm (12) suite.

  • Create Keyring & Set Permissions:
sudo mkdir -p /etc/apt/keyrings
sudo chmod 755 /etc/apt/keyrings

Download GPG Key: 

sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

Define Repository (The Trixie Override): Note: We explicitly use bookworm here instead of the dynamic $(VERSION_CODENAME) because the Trixie repository does not exist on Docker's servers yet. 

echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \ bookworm stable" | \  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Install Docker Engine: 

sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin
  • Prepare Storage on Mango: Since we have a 500GB data disk (/mnt/metrics_data), it’s a good idea to store our stack configurations there. This ensures that even if the OS drive (32GB) fills up or needs a wipe, our deployment YAMLs are safe. Create directories on your data disk and create a symlink so the paths look "standard"
sudo mkdir -p /mnt/metrics_data/dockge/data
sudo mkdir -p /mnt/metrics_data/dockge/stacks
sudo ln -s /mnt/metrics_data/dockge /opt/dockge
  • Deploy Master Dockge: Create a directory for the compose file and launch it:
mkdir -p ~/dockge-master && cd ~/dockge-master
nano compose.yaml

Paste this configuration: 

services:
 dockge:
   image: louislam/dockge:1
   restart: unless-stopped
   ports:
     - 5001:5001
   volumes:
     - /var/run/docker.sock:/var/run/docker.sock
     - /opt/dockge/data:/app/data
     - /opt/dockge/stacks:/opt/stacks
   environment:
     # Tell Dockge where to find the stacks on the HOST
     - DOCKGE_STACKS_DIR=/opt/dockge/stacks

Save & close then start it up: 

sudo docker compose up -d


The Amalgamation: Linking Quince, Blackberry, & Tayberry

The first thing to do is to allow the Dockge management port through Pfsense from production to Infra by creating an alias of port 5001 and creating the pass rule between the two VLANs. Once the firewall rule is enabled we login to th the Dockge on mango at http://mango:5001, assuming the three remote nodes are running, we will follow these steps to centralize our lab:

  • Click the "Add Agent" button on the Dockge Agents panel on the left of the Dockge screen
  • Fill in the text boxes with
  • click connect and wait for the target to login. If it times out check the username/password and check the Pfsense rule is allowing port 5001 from infra to production


Testing Master Dockge

To test and demonstrate the master Dockge management we can install a test application on each of the target nodes. For the test we will use the stack name test-log-viewer and the YAML 

services:
 dozzle:
   container_name: dozzle_test
   image: amir20/dozzle:latest
   restart: unless-stopped
   ports:
     - 8888:8080
   environment:
     - DOZZLE_LEVEL=info
   volumes:
     - /var/run/docker.sock:/var/run/docker.sock
  • Click + Compose
  • copy the stack name test-log-viewer
  • Use the drop down docker agent to select the remote target node for the application, there should be current(this is the local node mango), Blackberry, Tayberry and Quince all as online
  • Paste in the above YAML to the Compose.yaml
  • click deploy button and the target node should pull down the application and start
  • After proving that the application has been deployed it can be deleted


🔐 Vaultwarden Deployment on Mango

It is time to retire the KeePass file and as we have the new SSL certs and Dockge setup it is time to deploy Vaultwarden. Since our certificates are already synced to Mango, Vaultwarden will be able to start with full HTTPS right out of the gate. We will use the certificates located at /etc/nginx/ssl/seaoffate.net/. Even though they are owned by nigel, the Docker mount will allow Vaultwarden to read them perfectly. We will have backups on a daily schedule to Raisin. The deployment uses a "Full-Chain TLS" model. Encryption is maintained from the client browser all the way to the application container.

🔧Container Deployment (Dockge)

Vaultwarden is deployed via Docker Compose with the ROCKET_TLS feature enabled. This ensures the container itself handles encryption using the wild-card certificates synced from Raisin.

  • Open Mango Dockge http://mango:5001
  • Click + Compose and name the stack vaultwarden
  • Leave Dockge Agent as Current
  • Paste the following YAML:
services:
  vaultwarden:
    image: vaultwarden/server:latest
    container_name: vaultwarden
    restart: unless-stopped
    environment:
      - DOMAIN=https://vault.seaoffate.net
      - SIGNUPS_ALLOWED=true
      - ADMIN_TOKEN=yoursecret_change_this # Used for the /admin panel
      - ROCKET_TLS={certs="/ssl/fullchain.pem",key="/ssl/privkey.pem"}
      - ROCKET_PORT=80
    volumes:
      - ./data:/data
      - /etc/nginx/ssl/seaoffate.net/fullchain.pem:/ssl/fullchain.pem:ro
      - /etc/nginx/ssl/seaoffate.net/privkey.pem:/ssl/privkey.pem:ro
    ports:
      - 443:80
  • Click Deploy and the vault warden image will be downloaded

💡By enabling ROCKET_TLS with the public SSL Cert from Letencypt, in the Vaultwarden container on Mango, the container itself is expecting an encrypted connection and connection requests from within our home lab, inside the Pfsense VLANs will also be over TLS and no matter how the lap is reconfigured all passwords will always be passed over an encrypted channel.

🍇 Nginx Reverse Proxy Configuration (Raisin)

Raisin acts as the TLS gateway. It intercepts external requests and passes them to Mango over the internal network. Note We will not allow any http access on port 80 and the default site config elsewhere would drop any clear text connection attempt if any of the firewalls were inadvertently left forwarding port 80 on to Raisin.

  • Create a new configuration in the sites-available
sudo nano /etc/nginx/sites-available/vault.seaoffate.net
  • Paste the following text to the config
server {
    listen 443 ssl;
    server_name vault.seaoffate.net;
    # 
    ssl_certificate /etc/letsencrypt/live/seaoffate.net/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/seaoffate.net/privkey.pem;
    #
    location / {
        proxy_pass https://192.168.110.133:443;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        #
        # WebSockets for real-time sync
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}
  • Save & close the vault.seaoffate.net
  • Enable the site
sudo ln -s /etc/nginx/sites-available/vault.seaoffate.net /etc/nginx/sites-enabled/
  • Test the config:
sudo nginx -t
  • Reload Nginx & check the sevice:
sudo systemctl reload nginx
sudo systemctl status nginx

💡 Why we use https for the proxy_pass

Since we enabled ROCKET_TLS in the Vaultwarden container on Mango, the container itself is expecting an encrypted connection. By using proxy_pass https://..., Raisin and Mango will talk to each other over an encrypted link inside our network. With the current configuration both Raisin and Mango are physically on the same Proxmox host so very little chance that the communication could be intercepted. However, if at some point the VMs are moved it is be better to keep the channel encrypted end to end rather than as some sources suggest and pass the traffic via http on the backend. It has been noted that most guides that setup services with docker have a tendency to treat security as an after thought or a luxury, even with applications such as this where passwords are being accessed, by terminating the https at the reverse proxy. While that will indeed satisy the Cloudflare "full strict" requirement for end to end SSL, it is a big cheat and security hole that should be recognised as such & avoided. We prefer to use the SSL end to end and use the public cert so we get no browser warnings from within the LAN or outside.

📽️First use and Finalise Vaultwarden

As can be noted in the YAML file there are two lines that look suspicious the new accounts and the admin token. These are left as they are for the initial setup but we will change them now. first the new accounts needs to be remedied so that if someone on the Internet stumbles across our vault they don't start filling our password manager with bogus junk. Obviously, we need to create our own account so we logon to https://vault.seaoffate.net. At the vault warden logon screen create the account/s as required. Once all of the accounts are created we can change the line in the environment "SIGNUPS_ALLOWED=true" to SIGNUPS_ALLOWED=false and restart the application. From this point on no new accounts can be created, while the logon screen does continue to show the option to do so it will not actually complete the new account creation process.

Securing the Admin Logon

The next thing to address is the "ADMIN_TOKEN=yoursecret_change_this # Used for the /admin panel" line. The admin logon is accessed from the URL https://vault.seaoffate.net/admin and is be used to configure options for the Vault warden users. There is no username just a password. To avoid posting a password on the Dockge screen we use a password encryption routine supplied by the Vault warden application. to create a password follow these steps

  • Logon to the VM that is hosting the vault warden application (in this case Mango)
ssh nigel@mango
  • generate the password hash
docker exec -it vaultwarden /vaultwarden hash
  • Type in the password "simplepassword" (or one of your choice) and then confirm it
  • An admin token is displayed. ADMIN_TOKEN='$argon2id$v=19$m=65540,t=3,p=4$ddTHUmxG21CK2r3OHXblLVUokH/2qauOqpLp3+uHSms$4/UZtnSUM3U3dFzyvMVtyYFZgDlnPgqXnd79r0wGd+0'
  • Docker compose YAML treats $ as variables so makes a mess of the token. To fix the token copy the long string between the quotation marks and paste it in to notepad or similar and replace each $ with $$. eg
$argon2id$v=19$m=65540,t=3,p=4$ddTHUmxG21CK2r3OHXblLVUokH/2qauOqpLp3+uHSms$4/UZtnSUM3U3dFzyvMVtyYFZgDlnPgqXnd79r0wGd+0

becomes 

$$argon2id$$v=19$$m=65540,t=3,p=4$$ddTHUmxG21CK2r3OHXblLVUokH/2qauOqpLp3+uHSms$$4/UZtnSUM3U3dFzyvMVtyYFZgDlnPgqXnd79r0wGd+0
  • The final string can be pasted into the YAML line in the environment
- ADMIN_TOKEN=yoursecret_change_this # Used for the /admin panel

becomes 

- ADMIN_TOKEN=$$argon2id$$v=19$$m=65540,t=3,p=4$$ddTHUmxG21CK2r3OHXblLVUokH/2qauOqpLp3+uHSms$$4/UZtnSUM3U3dFzyvMVtyYFZgDlnPgqXnd79r0wGd+0 # Used for the /admin panel
  • Restart the docker application
  • http://vault.seaoffate.net/admin
  • Login with the password simplepassword (or whatever was created) in the earlier SSH session just to prove it works.

🚨 If the admin password is lost a new password must be created in the terminal and entered in the YAMl compose file.

🏛️Data Integrity & Backups

Backups are "Pulled" from Mango to Raisin nightly to ensure data redundancy. It should be noted that the SQL Lite file that stores the passwords will encrypt the passwords but not the usernames so it would be better if the backups are kept in a secure location for the purposes of the documentation the backups are assumed to be stored on Raisin but they could be stored anywhere that is most convent & secure with simple adjustments to the procedure.

  • From the terminal of Raisin we need a passwordless logon to mango for the backup script to run
ssh-copy-id [email protected]
  • If an error like "/usr/bin/ssh-copy-id: ERROR: No identities found" use the keygen routine (just accept the defaults)
ssh-keygen -t ed25519 -C "nigel@raisin"
  • Create the backup script and file locations
mkdir -p ~/scripts
mkdir -p ~/logs
nano ~/scripts/backup-vault.sh
  • Copy the script to the file
#!/bin/bash
#
# Configuration
REMOTE_USER="nigel"
REMOTE_HOST="mango.seaoffate.local"
REMOTE_DIR="/opt/dockge/stacks/vaultwarden/data"
BACKUP_DIR="/home/nigel/backups/vaultwarden"
TIMESTAMP=$(date +"%Y-%m-%d_%H-%M-%S")
#
# Create local backup directory if it doesn't exist
mkdir -p "$BACKUP_DIR"
#
echo "Starting Vaultwarden backup from Mango..."
#
# Sync files
if rsync -avz -e ssh "$REMOTE_USER@$REMOTE_HOST:$REMOTE_DIR" "$BACKUP_DIR/latest/"; then
    echo "Files synced successfully."
    #
    # Create the timestamped archive
    tar -czf "$BACKUP_DIR/vault_backup_$TIMESTAMP.tar.gz" -C "$BACKUP_DIR/latest" .
    #
    # Delete files older than 30 days
    find "$BACKUP_DIR" -name "vault_backup_*.tar.gz" -mtime +30 -delete
    #
    echo "Backup complete: $BACKUP_DIR/vault_backup_$TIMESTAMP.tar.gz"
else
    echo "ERROR: Rsync failed. No backup created."
    exit 1
fi
  • Save & close, then make it executable
chmod +x ~/scripts/backup-vault.sh
  • Make sure rsync is installed on both Raisin and Mango
  • on Raisin
sudo apt update && sudo apt install rsync -y
  • login to mango
ssh nigel@mango
  • install rsync & exit
sudo apt update && sudo apt install rsync -y
exit
  • When back on raisin we need to add a new crontab (choose option 1 if asked)
crontab -e
  • Add the line to the bottom of the list to run the script every morning at 03:00
# Every morning at 3:00 AM, pull the vault data from Mango
0 3 * * * /home/nigel/scripts/backup-vault.sh >> /home/nigel/logs/vault-backup.log 2>&1
  • save & close, we can test the script with
~/scripts/backup-vault.sh  
ls -lh ~/backups/vaultwarden

There should be at least one backup showing

Retrieved from "https://wiki.seaoffate.net/index.php?title=Unified_Monitoring_Stack&oldid=1304"