Pineapple

Introduction

Pineapple, at x.x.x.130 on the Infra network, is the host to the Prometheus application to gather metrics from each VM host and from Pear using agents installed on each host. The partner application, Grafana hosted on Granadilla is used to view the data collected by Prometheus.

Security concerns

The purpose of Prometheus is to gather data concerning all of the hosts on the network making it a good source of information to any hostile actor. Keeping it inside Infra and not publishing it's webserver to the Internet would be obvious security measures. Making specific aliases & rules on Pfsense for it to access it's agents would also be required actions (aliases for these obscure ports does make it a lot more secure and readable).

Prometheus Installation

The setup of Prometheus will have several separate parts.

• Server software installation
• Server configuration
• Firewall rules setup
• Agent installation

Prometheus Setup

The first thing was to create a VM in the Infra network and give it a hostname of Pineapple and IP/gateway (x.x.x.130/24) to match. To set the hostname & IP address just use the script but we must remember to edit the gateway address in /etc/netplan

sudo nano /etc/netplan/some_config_file.yaml
sudo netplan apply

We need to make sure that the host is also listed in dns by logon to ctns1 and using the add_combined_hostadd.sh. Then we do the ubiquitous

sudo apt update && sudo apt upgrade -y

We will need wget and tar if they are not already installed

sudo apt install -y wget tar

Next we have to make a user "prometheus" for the application to run as

sudo useradd --no-create-home --shell /bin/false prometheus

and make some dirs with the user as owner

sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

To download the Prometheus application we use wget but we have to locate the up to date file so browse to https://prometheus.io/download/ find the file prometheus-x.x.x.linux-amd64.tar.gz and copy the link address. Once we have the address we can wget it and extract it with the following command examples

wget prometheus-3.4.1.linux-amd64.tar.gz
tar -xvf prometheus-3.4.1.linux-amd64.tar.gz
cd prometheus-3.4.1.linux-amd64

Then copy the binaries to the relevant dirs and set permissions

sudo mv prometheus /usr/local/bin/
sudo mv promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool

Prometheus Configuration

The application is now installed so we can now configure it to scrape al of the target VMs with a yaml file that we will create.

sudo nano /etc/systemd/system/prometheus.service

The config file will look something like

global:
 scrape_interval: 15s # How frequently to scrape targets
 evaluation_interval: 15s # How frequently to evaluate rules

scrape_configs:
  # Prometheus monitoring itself (optional, but good for health checks)
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Node Exporters for your infrastructure VMs
  - job_name: 'node_exporter_infra'
    static_configs:
      - targets: ['x.x.x.x:9100', 'x.x.x.x:9100', 'x.x.x.x:9100'] # pineapple (Prometheus) and granadilla (Grafana) and ctns1 (dnsmasq)

  # Node Exporters for your production VMs (Webservers, Reverse Proxy, MySQL server if not using mysqld_exporter)
  - job_name: 'node_exporter_production'
    static_configs:
      - targets:
          - 'x.x.x.x:9100' # raisin Reverse Proxy nginx
          - 'x.x.x.x:9100' # Strawberry (backupserver)
          - 'x.x.x.x:9100' # plum  webserver (photo, wiki and www) apache2
          - 'x.x.x.x:9100' # satsuma (samba, photosort)
          - 'x.x.x.x:9100' # fig (nextcloud)
          - 'x.x.x.x:9100' # mandarin (Mysql)
          # Add other production VM IPs here as needed

  # Node Exporters for your VPN servers
  - job_name: 'node_exporter_vpn'
    static_configs:
      - targets:
          - 'x.x.x.x:9100' # Vanilla Wireguard VPN Server 
          - 'x.x.x.x:9100' # voavanga OpenVPN VPN server
          # Add other VPN server IPs here as needed

  # Node Exporters for your terminal VMs
  - job_name: 'node_exporter_terminals'
    static_configs:
      - targets:
          - 'x.x.x.x:9182'  # Wahoo Win 11 desktop
          - 'x.x.x.x:9182'  # Walnut Win 11 desktop (with jellyfin) 
          - 'x.x.x.x:9100'  # Lychee linux desktop
          # Add other terminal VM IPs here as needed

  # Node Exporters for your mgt network VMs (if any you want to monitor)
  - job_name: 'node_exporter_mgt'
    static_configs:
      - targets:
          - 'x.x.x.x:9100' # Lemon
          # Add other mgt VM IPs here as needed

  # Job for Nginx Exporter on Raisin (192.168.100.9)
  - job_name: 'nginx_reverse_proxy_raisin'
    static_configs:
      - targets: ['x.x.x.x:9113'] # Default port for nginx-exporter  

  # Job for MySQL Exporter on Mandarin (192.168.100.8)
  - job_name: 'mysql_server_mandarin'
    static_configs:
      - targets: ['x.x.x.x:9104'] # Default port for mysqld_exporter

  # job for Apache Exporter on webservers
  - job_name: 'apache_webservers'
    static_configs:
      - targets:
          - 'x.x.x.x:9117' # plum  webserver (photo, wiki and www) apache2
          - 'x.x.x.x:9117' # satsuma (samba, apache2, photosort)
          - 'x.x.x.x:9117' # fig (nextcloud)

   # Job for Proxmox Host
   - job_name: 'proxmox_host_pear'
     static_configs:
       - targets:
           - 'x.x.x.x:9100' # Replace with your Proxmox host's actual IP

At the end of the file there is a load of comments to give some guidance on how to write the config, it would be better to leave them in for future reference.

The Prometheus server application has a webserver component that can be viewed on port 9090 as shown in the scrape_configs: section above. As has been noted there is a security implication to Prometheus in that it is giving detailed information about the state of the whole network so with that in mind the Pfsense rule allowing access should be kept specifically to the MGT network. It will not make any difference to Grafana on Granadilla because it is on the the same network.

Pfsense Rules

Before we can see any data from Prometheus we will need to add the exporter agent to each machine and we will also need to add a rule to Pfsense to allow Prometheus to access the host being monitored, note the rule will be for Pineapple (Prometheus) on the Infra network to be the source and the host's network to be the destination because it is up to Prometheus to request the data, not the agent to send it. Assuming the above config we will need the following TCP rules

• On the Infra Interface allow source Pineapple port 9100 destination Production, MGT, VPNnet and Terminals port 9100. # This is the basic exporter
• On the Infra Interface allow source Pineapple port 9113 destination Production port 9113 # This is for Nginx specific exporter
• On the Infra Interface allow source Pineapple port 9117 destination Production port 9117 # This is for Apache specific exporter
• On the Infra Interface allow source Pineapple port 9104 destination Production port 9104 # This is for MySQL specific exporter
• On the Infra Interface allow source Pineapple port 9182 destination Terminals port 9182 # This is for Windows specific exporter
• On the Infra Interface allow source Pineapple port 9100 destination pear port 9100 # This is specifically to allow pineapple to access Pear and it will probably need to be on the WAN interface. Note that this rule is passing out of the network and onto the host Pear.
• On the MGT interface allow source MGT port 9090 destination Pineapple port 9090 # This rule is to allow lemon or any host on the MGT network to be able to view the Prometheus webserver on Pinapple port 9090

Agent Installation

When the rules are made to allow Prometheus to pull the data from it's agents we can start adding them to the VMs. We will install the node_exporter on everything as this is a basic CPU, RAM, Network ETC agent, the only exception is the two Windows 11 hosts. The other agents are specifically geared to a particular application so not required on every host.

Node Exporter

The basic agent to be installed on every Linux host. Start by adding a user to run the agent and a directory to put it.

sudo useradd --no-create-home --shell /bin/false node_exporter
sudo mkdir /etc/node_exporter

Then we need to locate the agent binaries for the most up to date version so browse to the github web page at https://prometheus.io/download/ and look for the version that says prometheus-x.x.x.linux-amd64.tar.gz and copy the link address, then we download it, uncompress it move it to the correct directory and set appropriate permissions.

wget https://github.com/prometheus/node_exporter/releases/download/v1.9.1/node_exporter-1.9.1.linux-amd64.tar.gz
tar -xvf node_exporter-1.9.1.linux-amd64.tar.gz 
cd node_exporter-1.9.1.linux-amd64/
sudo mv node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

We will need it to be a service so we need a service file

sudo nano /etc/systemd/system/node_exporter.service

and we need to add some boilerplate code to the service config

[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
    --web.listen-address="0.0.0.0:9100"

[Install]
WantedBy=multi-user.target

When we have saved and exited from the config we will need to Reload Systemd

sudo systemctl daemon-reload

Next we should enable, start and check the service

sudo systemctl enable node_exporter
sudo systemctl start node_exporter
sudo systemctl status node_exporter

To test that it is working we need to go back to pineapple and attemt to extract the data from the agent with the curl command

 curl http://x.x.x.x:9100/metrics

We should see a load of data flow on to the screen. If nothing is displayed there is a problem with either the firewall or the agent. The easiest way to isolate the problem to one or the other is to go back to the client where the agent was installed and run

 curl http://127.0.0.1:9100/metrics

As this is the local host we should see output from the agent. If we see output we need to check firewall/s both on the localhost and on Pfsense because clearly the agent is doing it's stuff but Pineapple cant read it. If there is no output we can be reasonably sure that the agent is not working.

As soon as curl on Pineapple starts returning data from the agent it will trigger the webserver on part of Prometheus to show the host as up