Pineapple - Revision history

Wikisailor: /* Introduction */

2026-01-27T14:52:04Z

Introduction

← Older revision		Revision as of 14:52, 27 January 2026
Line 1:		Line 1:
	==Introduction==		==Introduction==

	Pineapple, at x.x.x.130 on the Infra network, is the host to the Prometheus application to gather metrics from ~~each VM host~~ and from Pear using agents installed on each host. The partner application, Grafana hosted on '''[[Granadilla]]''' is used to view the data collected by Prometheus. An overview of the facilities offered by a '''[[Prometheus & Grafana ]]''' partnership can be found '''[[Prometheus & Grafana \| here]]'''.		Pineapple, at x.x.x.130 on the Infra network, is the host to the Prometheus application to gather metrics from the '''[[Virtual Machines]]''' and from Pear using agents installed on each host. The partner application, Grafana hosted on '''[[Granadilla]]''' is used to view the data collected by Prometheus. An overview of the facilities offered by a '''[[Prometheus & Grafana ]]''' partnership can be found '''[[Prometheus & Grafana \| here]]'''.
	===Security concerns===		===Security concerns===
	The purpose of Prometheus is to gather data concerning all of the hosts on the network making it a good source of information to any hostile actor. Keeping it inside Infra and not publishing it's webserver to the Internet would be obvious security measures. Making specific aliases & rules on Pfsense for it to access it's agents would also be required actions (aliases for these obscure ports does make it a lot more secure and readable).		The purpose of Prometheus is to gather data concerning all of the hosts on the network making it a good source of information to any hostile actor. Keeping it inside Infra and not publishing it's webserver to the Internet would be obvious security measures. Making specific aliases & rules on Pfsense for it to access it's agents would also be required actions (aliases for these obscure ports does make it a lot more secure and readable).

Wikisailor: /* Introduction */

2025-06-05T10:10:56Z

Introduction

← Older revision		Revision as of 10:10, 5 June 2025
Line 1:		Line 1:
	==Introduction==		==Introduction==

	Pineapple, at x.x.x.130 on the Infra network, is the host to the Prometheus application to gather metrics from each VM host and from Pear using agents installed on each host. The partner application, Grafana hosted on '''[[Granadilla]]''' is used to view the data collected by Prometheus. An overview of the facilities offered by '''[[~~Prometheus & Grafana \| a~~ Prometheus & Grafana ]]''' can be found '''[[Prometheus & Grafana \| here]]'''.		Pineapple, at x.x.x.130 on the Infra network, is the host to the Prometheus application to gather metrics from each VM host and from Pear using agents installed on each host. The partner application, Grafana hosted on '''[[Granadilla]]''' is used to view the data collected by Prometheus. An overview of the facilities offered by a '''[[Prometheus & Grafana ]]''' partnership can be found '''[[Prometheus & Grafana \| here]]'''.
	===Security concerns===		===Security concerns===
	The purpose of Prometheus is to gather data concerning all of the hosts on the network making it a good source of information to any hostile actor. Keeping it inside Infra and not publishing it's webserver to the Internet would be obvious security measures. Making specific aliases & rules on Pfsense for it to access it's agents would also be required actions (aliases for these obscure ports does make it a lot more secure and readable).		The purpose of Prometheus is to gather data concerning all of the hosts on the network making it a good source of information to any hostile actor. Keeping it inside Infra and not publishing it's webserver to the Internet would be obvious security measures. Making specific aliases & rules on Pfsense for it to access it's agents would also be required actions (aliases for these obscure ports does make it a lot more secure and readable).

Wikisailor: /* Introduction */

2025-06-05T10:09:49Z

Introduction

← Older revision		Revision as of 10:09, 5 June 2025
Line 1:		Line 1:
	==Introduction==		==Introduction==

	Pineapple, at x.x.x.130 on the Infra network, is the host to the Prometheus application to gather metrics from each VM host and from Pear using agents installed on each host. The partner application, Grafana hosted on '''[[Granadilla]]''' is used to view the data collected by Prometheus.		Pineapple, at x.x.x.130 on the Infra network, is the host to the Prometheus application to gather metrics from each VM host and from Pear using agents installed on each host. The partner application, Grafana hosted on '''[[Granadilla]]''' is used to view the data collected by Prometheus. An overview of the facilities offered by '''[[Prometheus & Grafana \| a Prometheus & Grafana ]]''' can be found '''[[Prometheus & Grafana \| here]]'''.
	===Security concerns===		===Security concerns===
	The purpose of Prometheus is to gather data concerning all of the hosts on the network making it a good source of information to any hostile actor. Keeping it inside Infra and not publishing it's webserver to the Internet would be obvious security measures. Making specific aliases & rules on Pfsense for it to access it's agents would also be required actions (aliases for these obscure ports does make it a lot more secure and readable).		The purpose of Prometheus is to gather data concerning all of the hosts on the network making it a good source of information to any hostile actor. Keeping it inside Infra and not publishing it's webserver to the Internet would be obvious security measures. Making specific aliases & rules on Pfsense for it to access it's agents would also be required actions (aliases for these obscure ports does make it a lot more secure and readable).

	==Prometheus Installation==		==Prometheus Installation==

Wikisailor: /* smartctl_exporter_script.sh (Proxmox Disk SMART Metrics) */

2025-06-05T01:15:18Z

smartctl_exporter_script.sh (Proxmox Disk SMART Metrics)

← Older revision		Revision as of 01:15, 5 June 2025
Line 682:		Line 682:
	* Add Grafana Dashboards Import Node - ZFS Stats (ID 7968) and S.M.A.R.T. (ID 22604)with x.x.x.x:9100 (pear's IP) as the instance.		* Add Grafana Dashboards Import Node - ZFS Stats (ID 7968) and S.M.A.R.T. (ID 22604)with x.x.x.x:9100 (pear's IP) as the instance.

	This was a major installation that did take a lot of time to install and troubleshoot all of the agents but it is done now. The next steps would be either to create rules & alerts in the Prometheus Web GUI or to install Grafana on '''[[Granadilla]]'''.		This was a major installation that did take a lot of time to install and troubleshoot all of the agents but it is done now. The next steps would be either to create rules & alerts in the Prometheus Web GUI or to install Grafana on '''[[Granadilla]]'''. There is a a dedicated Proxmox exporter that will be worth looking at

Wikisailor: /* smartctl_exporter_script.sh (Proxmox Disk SMART Metrics) */

2025-06-05T01:10:20Z

smartctl_exporter_script.sh (Proxmox Disk SMART Metrics)

← Older revision		Revision as of 01:10, 5 June 2025
Line 667:		Line 667:
	** Ensure the script (smartctl_exporter_script.sh) has execute permissions (chmod +x /usr/local/bin/smartctl_exporter_script.sh).		** Ensure the script (smartctl_exporter_script.sh) has execute permissions (chmod +x /usr/local/bin/smartctl_exporter_script.sh).
	** Cron jobs run in a very minimal environment. If your script relies on specific environment variables (e.g., HOME, LANG, PATH), you might need to set them explicitly within the script itself or at the top of your crontab file.		** Cron jobs run in a very minimal environment. If your script relies on specific environment variables (e.g., HOME, LANG, PATH), you might need to set them explicitly within the script itself or at the top of your crontab file.

			=====Post-Installation Verification Steps=====
			* Manual Test (once script is created):
			** Test as node_exporter user:
			su -c "bash /usr/local/bin/smartctl_exporter_script.sh" node_exporter
			** Check log file (should show successful execution, no errors) :
			cat /var/log/node_exporter/smartctl_exporter.log
			** Check metrics file (should show formatted SMART metrics for all drives) :
			cat /var/lib/node_exporter/textfile_collector/smart.prom
			* Prometheus Checks
			** After the script runs and node_exporter is restarted, check Prometheus UI directly for node_zfs_ and smartctl_ metrics.
			http://x.x.x.x:9100/metrics \| grep zfs
			** Confirm proxmox_host_pear target is UP in Prometheus UI http://pineapple:9090
			* Add Grafana Dashboards Import Node - ZFS Stats (ID 7968) and S.M.A.R.T. (ID 22604)with x.x.x.x:9100 (pear's IP) as the instance.

			This was a major installation that did take a lot of time to install and troubleshoot all of the agents but it is done now. The next steps would be either to create rules & alerts in the Prometheus Web GUI or to install Grafana on '''[[Granadilla]]'''.

Wikisailor: /* smartctl_exporter_script.sh (Proxmox Disk SMART Metrics) */

2025-06-05T00:24:23Z

smartctl_exporter_script.sh (Proxmox Disk SMART Metrics)

← Older revision		Revision as of 00:24, 5 June 2025
Line 646:		Line 646:
	exit 0		exit 0
	</syntaxhighlight>		</syntaxhighlight>
			Make the Script Executable:
			chmod +x /usr/local/bin/smartctl_exporter_script.sh
			Schedule the Script with Cron to run the script every 12 hours (There does not seem much point in having the second by second status of SMART as it is unlikely to change particularly quickly, so twice per day seems to be more than adequate). To pen the crontab for the node_exporter user:
			crontab -e -u node_exporter
			and add
			0 /12 * * /usr/local/bin/smartctl_exporter_script.sh > /dev/null 2>&1
			The contab entry has six fields: Minute (0-59), Hour (0-23, where 0 is midnight), Day of Month (1-31), Month (1-12 or Jan-Dec),Day of Week (0-7, where 0 and 7 are Sunday) and finally the command to execute. So that means this command will run
			* 0 (First field: Minute): "at the 0th minute of the hour" (i.e., on the hour).
			* */12 (Second field: Hour):
			** The * means "every possible value" for that field.
			** The / indicates a step value.
			** The 12 combined with the last / means every 12 hours
			* The * in the third field (day of the Month) means every day
			* The next * in the forth field (Month) means every month
			* The * in the fifth field (day of the week) means everyday
			* The sixth and final field is the path to the executable file to be run.
			** It is always the best practice to have the path as an absolute value and not as a relative path like ~/some_script.sh.
			** It is better to have the script handle it's own logging.
			** Output from cron jobs (to stdout or stderr) will typically be emailed to the user who owns the crontab entry so if we don't want emails we can suppress all output by adding the > /dev/null 2>&1 to the end of the options so all output will be redirected to the null device. If we wanted output to go to a log file we would append it with the suffix >> /var/log/smartctl_exporter.log 2>&1
			** Ensure the script (smartctl_exporter_script.sh) has execute permissions (chmod +x /usr/local/bin/smartctl_exporter_script.sh).
			** Cron jobs run in a very minimal environment. If your script relies on specific environment variables (e.g., HOME, LANG, PATH), you might need to set them explicitly within the script itself or at the top of your crontab file.

Wikisailor: /* smartctl_exporter_script.sh (Proxmox Disk SMART Metrics) */

2025-06-04T23:48:34Z

smartctl_exporter_script.sh (Proxmox Disk SMART Metrics)

← Older revision		Revision as of 23:48, 4 June 2025
Line 473:		Line 473:
	* Enable node_exporter Collectors for S.M.A.R.T. and ZFS: The node_exporter needs to be told to activate its zfs collector (for ZFS pool statistics) and its textfile collector (which will read the smart.prom file generated by our script). Edit the node_exporter Systemd service file :		* Enable node_exporter Collectors for S.M.A.R.T. and ZFS: The node_exporter needs to be told to activate its zfs collector (for ZFS pool statistics) and its textfile collector (which will read the smart.prom file generated by our script). Edit the node_exporter Systemd service file :
	nano /etc/systemd/system/node_exporter.service		nano /etc/systemd/system/node_exporter.service
	Locate the ExecStart line and ensure it includes these flags:		Locate the ExecStart line and ensure it includes these flags:
			<syntaxhighlight lang="yaml">
	[Service]		[Service]
	User=node_exporter		User=node_exporter
Line 485:		Line 486:
	[Install]		[Install]
	WantedBy=multi-user.target		WantedBy=multi-user.target
			</syntaxhighlight>
	save & exit. Restart node_exporter service: This applies the new collector settings.		save & exit. Restart node_exporter service: This applies the new collector settings.
	systemctl restart node_exporter		systemctl restart node_exporter

Wikisailor: /* Prometheus Configuration */

2025-06-04T23:45:49Z

Prometheus Configuration

← Older revision		Revision as of 23:45, 4 June 2025
Line 43:		Line 43:
	sudo nano /etc/systemd/system/prometheus.service		sudo nano /etc/systemd/system/prometheus.service
	The config file will look something like		The config file will look something like
			<syntaxhighlight lang="yaml">
	global:		global:
	scrape_interval: 15s # How frequently to scrape targets		scrape_interval: 15s # How frequently to scrape targets
Line 117:		Line 118:
	- targets:		- targets:
	- 'x.x.x.x:9100' # Replace with your Proxmox host's actual IP		- 'x.x.x.x:9100' # Replace with your Proxmox host's actual IP
			</syntaxhighlight>
	At the end of the file there is a load of comments to give some guidance on how to write the config, it would be better to leave them in for future reference.		At the end of the file there is a load of comments to give some guidance on how to write the config, it would be better to leave them in for future reference.

Wikisailor: /* Node Exporter */

2025-06-04T23:44:42Z

Node Exporter

← Older revision		Revision as of 23:44, 4 June 2025
Line 150:		Line 150:
	sudo nano /etc/systemd/system/node_exporter.service		sudo nano /etc/systemd/system/node_exporter.service
	and we need to add some boilerplate code to the service config		and we need to add some boilerplate code to the service config
			<syntaxhighlight lang="yaml">
	[Unit]		[Unit]
	Description=Prometheus Node Exporter		Description=Prometheus Node Exporter
Line 165:		Line 165:
	[Install]		[Install]
	WantedBy=multi-user.target		WantedBy=multi-user.target
			</syntaxhighlight>
	When we have saved and exited from the config we will need to Reload Systemd		When we have saved and exited from the config we will need to Reload Systemd
	sudo systemctl daemon-reload		sudo systemctl daemon-reload

Wikisailor: /* Mysql Exporter on Mandarin */

2025-06-04T23:43:54Z

Mysql Exporter on Mandarin

← Older revision		Revision as of 23:43, 4 June 2025
Line 219:		Line 219:
	sudo nano /etc/systemd/system/mysqld_exporter.service		sudo nano /etc/systemd/system/mysqld_exporter.service
	With the following configuration		With the following configuration
			<syntaxhighlight lang="yaml">
	[Unit]		[Unit]
	Description=Prometheus MySQL Exporter		Description=Prometheus MySQL Exporter
Line 235:		Line 235:
	[Install]		[Install]
	WantedBy=multi-user.target		WantedBy=multi-user.target
			</syntaxhighlight>
	After we save and exit we reload the systemd and start the service		After we save and exit we reload the systemd and start the service
	sudo systemctl daemon-reload		sudo systemctl daemon-reload