Pineapple

Introduction

Pineapple, at x.x.x.130 on the Infra network, is the host to the Prometheus application to gather metrics from each VM host and from Pear using agents installed on each host. The partner application, Grafana hosted on Granadilla is used to view the data collected by Prometheus.

Security concerns

The purpose of Prometheus is to gather data concerning all of the hosts on the network making it a good source of information to any hostile actor. Keeping it inside Infra and not publishing it's webserver to the Internet would be obvious security measures. Making specific aliases & rules on Pfsense for it to access it's agents would also be required actions (aliases for these obscure ports does make it a lot more secure and readable).

Prometheus Installation

The first thing was to create a VM in the Infra network and give it a hostname of Pineapple and IP/gateway (x.x.x.130/24) to match. To set the hostname & IP address just use the script but we must remember to edit the gateway address in /etc/netplan

sudo nano /etc/netplan/some_config_file.yaml
sudo netplan apply

We need to make sure that the host is also listed in dns by logon to ctns1 and using the add_combined_hostadd.sh. Then we do the ubiquitous

sudo apt update && sudo apt upgrade -y

We will need wget and tar if they are not already installed

sudo apt install -y wget tar

Next we have to make a user "prometheus" for the application to run as

sudo useradd --no-create-home --shell /bin/false prometheus

and make some dirs with the user as owner

sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

To download the Prometheus application we use wget but we have to locate the up to date file so browse to https://prometheus.io/download/ find the file prometheus-x.x.x.linux-amd64.tar.gz and copy the link address. Once we have the address we can wget it and extract it with the following command examples

wget prometheus-3.4.1.linux-amd64.tar.gz
tar -xvf prometheus-3.4.1.linux-amd64.tar.gz
cd prometheus-3.4.1.linux-amd64

Then copy the binaries to the relevant dirs and set permissions

sudo mv prometheus /usr/local/bin/
sudo mv promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus

sudo chown prometheus:prometheus /usr/local/bin/promtool

The application is now installed so we can now configure it to scrape al of the target VMs with a yaml file that we will create.

sudo nano /etc/systemd/system/prometheus.service

The config file will look something like

global:
 scrape_interval: 15s # How frequently to scrape targets
 evaluation_interval: 15s # How frequently to evaluate rules

scrape_configs:
  # Prometheus monitoring itself (optional, but good for health checks)
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Node Exporters for your infrastructure VMs
  - job_name: 'node_exporter_infra'
    static_configs:
      - targets: ['x.x.x.x:9100', 'x.x.x.x:9100', 'x.x.x.x:9100'] # pineapple (Prometheus) and granadilla (Grafana) and ctns1 (dnsmasq)

  # Node Exporters for your production VMs (Webservers, Reverse Proxy, MySQL server if not using mysqld_exporter)
  - job_name: 'node_exporter_production'
    static_configs:
      - targets:
          - 'x.x.x.x:9100' # raisin Reverse Proxy nginx
          - 'x.x.x.x:9100' # Strawberry (backupserver)
          - 'x.x.x.x:9100' # plum  webserver (photo, wiki and www) apache2
          - 'x.x.x.x:9100' # satsuma (samba, photosort)
          - 'x.x.x.x:9100' # fig (nextcloud)
          - 'x.x.x.x:9100' # mandarin (Mysql)
          # Add other production VM IPs here as needed

  # Node Exporters for your VPN servers
  - job_name: 'node_exporter_vpn'
    static_configs:
      - targets:
          - 'x.x.x.x:9100' # Vanilla Wireguard VPN Server 
          - 'x.x.x.x:9100' # voavanga OpenVPN VPN server
          # Add other VPN server IPs here as needed

  # Node Exporters for your terminal VMs
  - job_name: 'node_exporter_terminals'
    static_configs:
      - targets:
          - 'x.x.x.x:9182'  # Wahoo Win 11 desktop
          - 'x.x.x.x:9182'  # Walnut Win 11 desktop (with jellyfin) 
          - 'x.x.x.x:9100'  # Lychee linux desktop
          # Add other terminal VM IPs here as needed

  # Node Exporters for your mgt network VMs (if any you want to monitor)
  - job_name: 'node_exporter_mgt'
    static_configs:
      - targets:
          - 'x.x.x.x:9100' # Lemon
          # Add other mgt VM IPs here as needed

  # Job for Nginx Exporter on Raisin (192.168.100.9)
  - job_name: 'nginx_reverse_proxy_raisin'
    static_configs:
      - targets: ['x.x.x.x:9113'] # Default port for nginx-exporter  

  # Job for MySQL Exporter on Mandarin (192.168.100.8)
  - job_name: 'mysql_server_mandarin'
    static_configs:
      - targets: ['x.x.x.x:9104'] # Default port for mysqld_exporter

  # job for Apache Exporter on webservers
  - job_name: 'apache_webservers'
    static_configs:
      - targets:
          - 'x.x.x.x:9117' # plum  webserver (photo, wiki and www) apache2
          - 'x.x.x.x:9117' # satsuma (samba, apache2, photosort)
          - 'x.x.x.x:9117' # fig (nextcloud)

   # Job for Proxmox Host
   - job_name: 'proxmox_host_pear'
     static_configs:
       - targets:
           - 'x.x.x.x:9100' # Replace with your Proxmox host's actual IP