Unified Monitoring Stack

📖Introduction

Mango, located at 192.168.110.133 on the Infra network, is the unified successor to the Prometheus & Grafana and Victoria triad. It serves as the central hub for the Home Lab's observability. Mango natively scrapes metrics from all Virtual Machines, the Proxmox host(Pear) and the services, stores them in a high-performance VictoriaMetrics time-series database, and provides a Grafana interface for visualization.

By consolidating these services, we reduce network overhead and simplify the management of our monitoring infrastructure while maintaining 12-month data retention on a dedicated 500GB storage pool.

🚦Security & Network Architecture

Mango sits within the Infra network. Because it aggregates data from every host in the lab, it is a high-value target.

• Web Interfaces: Grafana (Port 3000) and VictoriaMetrics VMUI (Port 8428) are restricted via pfSense to be accessible only from the MGT network (Cinnamon/Lemon).
• Scraping Flow: Mango acts as the source for all scrape requests. pfSense rules must allow Mango to reach out to Production, VPN, and Terminal networks on specific exporter ports (9100, 9113, 9117, etc.).
• Storage Pool: Data is stored on a dedicated 500GB virtual disk (PearPool), mounted at /mnt/metrics_data to ensure that metric growth never impacts the OS root partition.

🏛️Environment & Storage Setup

The VM was created using the Debian Gold Master template.

• Hostname: Mango
• IP/Gateway: 192.168.110.133 / 192.168.110.1
• Disk 1 (OS): 32GB
• Disk 2 (Data): 500GB (Added via Proxmox)

Storage Initialization To handle the long-term metrics, the 500GB disk was initialized and mounted:

# Identify disk (sdb), format, and mount
sudo mkfs.ext4 /dev/sdb
sudo mkdir -p /mnt/metrics_data
sudo mount /dev/sdb /mnt/metrics_data
# Ensure persistence in /etc/fstab
/dev/sdb  /mnt/metrics_data  ext4  defaults  0  2

🔧Installation

⚡VictoriaMetrics Installation

VictoriaMetrics was installed as a native binary (not Docker) to replace both the Prometheus scraper and the Victoria storage VM.

• User & Directory Setup

sudo useradd --no-create-home --shell /bin/false victoriametrics
sudo mkdir /etc/victoriametrics
sudo chown -R victoriametrics:victoriametrics /etc/victoriametrics /mnt/metrics_data

• Binary Installation

Binaries were retrieved from the VictoriaMetrics GitHub.

wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.xx.x/victoria-metrics-linux-amd64-v1.xx.x.tar.gz
tar -xvf victoria-metrics-linux-amd64-v1.xx.x.tar.gz
sudo mv victoria-metrics-prod /usr/local/bin/victoriametrics
sudo chown victoriametrics:victoriametrics /usr/local/bin/victoriametrics

• Service Configuration

sudo nano /etc/systemd/system/victoriametrics.service

[Service]
ExecStart=/usr/local/bin/victoriametrics \
  --storageDataPath=/mnt/metrics_data \
  --retentionPeriod=12 \
  --promscrape.config=/etc/victoriametrics/prometheus.yml \
  --httpListenAddr=0.0.0.0:8428

Note: The --retentionPeriod=12 ensures one year of history.

🔍Scraping Configuration (prometheus.yml)

VictoriaMetrics uses the standard Prometheus YAML format for its scraper. The file was created with

sudo nano /etc/victoriametrics/prometheus.yml

Key Change: The evaluation_interval directive was removed as it is not natively supported by the VictoriaMetrics single-binary scraper (it expects vmalert for that).

🧪Target Jobs

The configuration includes the legacy fleet plus the new 2026 additions:

• Infrastructure: Mango (Self), CTNS1.
• Production:
  • Reverse proxy (Nginx) Raisin
  • Webservers (Apache) Plum, Satsuma, Fig
  • Database server (MySQL) Mandarin
• New 2026 Hosts: Blackcurrant (Data & Archive), Quince (AI/Media), Tayberry (OpenAlex).
• Gaming: Apple & Cherry (Minecraft Servers).
• Terminals:
  • (NoMachine) Kiwiberry
  • (XRDP), Kapok
  • (Windows, RDP) Wahoo/Walnut .

Scrape Interval: Set to 120s to balance data resolution with disk I/O and longevity.

Grafana Installation

Grafana was installed on the same Mango host to provide the local visualization layer.

• Repository & App Setup

sudo apt install -y apt-transport-https software-properties-common wget
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /usr/share/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/usr/share/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update && sudo apt install grafana -y
sudo systemctl enable --now grafana-server

🔀Network & Firewall Rules (pfSense)

To allow the new ports to function, pfSense was updated with a new Alias for Monitoring Ports:

• 3000: Grafana UI (remains the same from the previous Grafana installation)
• 8428: VictoriaMetrics UI/API ( The new port added for viewing of the scraping progress as was done by Prometheus web gui)
• 9090: removed the older Prometheus webgui port

Critical Rules

• MGT -> Mango: Allow ports 3000 & 8428 (Access from Cinnamon or other management console).
• Mango -> All Networks:
  • Allow port 9100 (Node)
  • Allow port 9113 (Nginx)
  • Allow port 9117 (Apache)
  • Allow port 9104 (MySQL)
  • Allow port 9182 (Windows)