Proxmox Server

From Sea of Fate
Jump to navigationJump to search

📋Introduction

This is the server that will sit under my desk until I move house. It has the hostname Pear and will be the beginning of the Home Lab. There is also now a new smaller and less powerful server with the hostname Kiwi.

🥝Create Virtual Machine from a template

it is possible to create a new VM based on a fixed template. The two possibilities are a full clone or a linked clone. the linked clone is the preferred as it should need less storage but the full clone would be completely independent of the template so it will use more storage and take longer to create.We can go down that rabbit hole when we look ant Ansible in a big way. For now just understand that we can call a new vm using the qm clone command. the will be more details to follow.

Orchard (Pear)

The first host has Proxmox installed with the following specification

🛠️Hardware Specification

The Main components were purchased in 2024 as follows

  • The Cpu is AMD Ryzen 9 5950X Processor (16Cores/32Threads, 105W TDP, Socket AM4, 72MB Cache, Up to 4.9 GHz Max Boost, no cooler) £369.98
  • The cooler is Noctua NH-D15, Premium CPU Cooler with 2x NF-A15 PWM 140mm Fans (Brown) £99.95
  • Mainboard is Gigabyte B550M AORUS ELITE Motherboard - Supports AMD Ryzen 5000 Series AM4 CPUs, 5+3 Phases Pure Digital VRM, up to 4733MHz DDR4 (OC), 2xPCIe 3.0 M.2, GbE LAN, USB 3.2 Gen1 £103.48
  • 128gb Ram is configured as two sets of CORSAIR VENGEANCE LPX DDR4 RAM 64GB (2x32GB) 3200MHz CL16-20-20-38 1.35V (CMK64GX4M2E3200C16) £112.20 for one set
  • GPU is a Gigabyte Geforce RTX 5060 TI with 16 GB Ram
  • Power supply is Corsair RM750x 80 Plus Gold Fully Modular ATX 750 Watt Power Supply £109.00
  • The case is Fractal Design Node 804 - Black - Compact Computer Case - mATX - High Airflow - Modular interior - 3x Fractal Design Silent R2 120mm Fans Included - Water-cooling ready - USB 3.0 -Window Side Panel £99.95
  • SSD is Crucial P3 Plus SSD 4TB M.2 NVMe PCIe Gen4 Internal SSD, Up to 4800MB/s, Laptop & Desktop (PC) Compatible, Solid State Drive - CT4000P3PSSD801 £189.99 and £159.99
  • Three 16 TB Seagate Ironwolf NAS drives make up the hard disk storage at a cost of £179.94 each total £ 539.82.
  • One cheap 1 TB SATA SSD currently set as a L2ARC but may reconfigure as a sacrificial drive for the high wear VMs if it looks like it is not getting many hits as L2ARC Update no longer used as L2ARC and formmated to a zpool fastpool.

Configuration

🌐Network Access

The IP Address and port of the host is 192.168.0.110:8006. I have my virgin media router forwarding all incoming traffic to 192.168.0.125 so it will go directly to the Pfsense firewall. All of the terminals can also send traffic to the WAN port of Pfsense. Edit with the change to Vodaphone the network address changed to 192.168.1.0/24 so Proxmox console is now 192.168.1.111:8006 and the WAN port of Pfsense is 192.168.1.125. a further change was to only forward those ports that are actually needed to Pfsense because the new ISP router has better options than the really old Virgin media one ( to be fair if i had stayed with Virgin i would have got a newer router). More Network Configuration details.

💾Storage

The main storage is the three 16TB hard drives configure into ZFS z1 so that they have one redundant disk. As one disk is redundant, obviously, it has 32TB of storage available.

I couldn't get PCI passthrough to work reliably without significant effort and some expense so I set up ZFS on Proxmox itself or more precisely on the Debian Linux that Proxmox resides on. Unfortunately, that means that I have to do any config on the CLI. I may write some scripts to perform some of the admin tasks or more likely setup a Nginx container to automate as much of it as possible. For the time being some useful ZFS commands can be found here

🥝 Grove (Kiwi) : Technical Manual & Build Log

Kiwi is the secondary Proxmox host in the Home Lab, housed in a Fractal Design ITX chassis. While less powerful than the primary host (Pear), it serves as a high-capacity storage node and 2.5G data target.

Hardware Specification

  • CPU: AMD Ryzen 5 with Stock Wraith Cooler.
  • RAM: 64GB (2x32GB Crucial DDR4), maxing out the two available slots.
  • System Drive: 2TB Samsung 990 Pro NVMe (Firmware verified: 5B2QJXD7).
  • Storage Pool: 3x 14TB Seagate IronWolf Pro HDDs (ZFS RAIDZ1, 28TB usable).
  • Network: Onboard 2.5 Gbps Wired NIC (nic0) + Built-in Wi-Fi (wlp5s0).
  • GPU: Legacy Nvidia GPU (Required for POST and initial installation). The Mainboard has 4 SATA ports on board. Three of them have 14 TB Ironwolf pro drives to be formatted as a ZFS array with Z! so will give 28TB of usable storage with one disk of redundancy

Proxmox Installation

It is assumed that the Proxmox installer has been downloaded from their website and the iso has been copied on to a USB stick and made bootable with Rufus or some other similar. If the reader doesn't know how to make a USB installer from an ISO try typing "How do I make a USB stick into a proxmox installer" in to a browser. there will be loads of guides on how to use Rufus. When a bootable USB stick with Proxmox installer on it has been created insert in to one of the USB slots along with a keyboard, mouse and screen. Please note the keyboard mouse and screen will only be needed for the installation, once Proxmox is installed it is unlikely that these will ever be used again because virtually all administration is performed on a web GUI.

BIOS settings for Hypervisors (including Proxmox)

Generally most BIOS can be left at defaults and a PC as a PC will work well enough, maybe the boot priority may need to be changed from a HDD to a SSD but even that is unlikely. It is possible to optimise lots of things in BIOS and for some people that is an option but it is also easy to make mistakes that either stop the machine from working or making the machine unstable so for most people, most of the time, it is better to leave BIOS alone. A big exception is when a hypervisor is to be install because a few options that are off by default need to be set for Virtualisation to work properly. The main things to set are:-

  • SVM Mode (AMD-V): Must be Enabled. This is what allows Proxmox to run VMs. Without it virtualisation will fail with most or possibly all hypervisors
  • IOMMU: Set to Enabled. This is vital if you ever want to "pass through" a SATA controller or GPU directly to a VM. while not vital for the virtualisation itself most hypervisors have this enabled.
  • Power Supply Idle Control: If you find the server randomly freezes when idle, set this to "Typical Current Idle". Ryzen chips sometimes drop voltage too low for some power supplies when idling, causing a crash. Again not vital but a "nice to have" if possible in the BIOS.
  • Boot options: The SSD should be at the top. It is better to set the USB at the bottom so that if a USB is left in the machine and it is rebooted it doesn't try to boot from the USB.

Once the BIOS option are set save the settings and quit. The computer will shutdown and restart, while it is restarting keep pressing the delete key or the F12 key to bring up the boot options (it is usually delete key but it should also display the options briefly on the screen as it starts).

GPU Considerations

The Proxmox installation is a little bit more tricky than is standard for the majority of computers in that this has an old GPU that the installer does not recognise. A GPU is required so that the machine will pass POST but is not really need or used by the Proxmox guests so is generally of little benefit for the overall Proxmox day to day running. A lot of Hypervisor hosts have the cheapest and lowest power GPUs installed just enough to POST but no more than that. The reason that GPU is of low priorty for most hypervisors hosts is that they are generally run headless, that is without a monitor attached, all interations with the host is through a dedicated application like Citrix XEN, or more likely through a web browser like Proxmox. Web based GUIs are popular because it is a lot easier and lighter processor load to have a webserver attached than it is to design a whole application. Kiwi has an old Nvidia GPU that works in that it provides a display but that is about it. Unfortunately, it does give some problems for the installer as it is so old but therea re steps to overcome that as will be detailed below. We may add a bigger and better GPU at a later date but for the foreseeable future that is in the realms of an upgrade for later and dependant on inheriting a GPU from another box.

Proxmox Installer problems (The Kiwi exceptions)

As stated earlier the Proxmox installer does not work with the GPU. This is shown in that the machine boots and loads the first screen of the installer with welcome to.. but when the first option is tried it starts to load then freezes at the loading Nvidia drivers or some such thing(cant remember exactly what it says). so to fix that issue we need to restart the host and when we get to the welcome screen again we highlight "Install Proxmox VE (Graphical)" but do not press Enter instead we press "e" so a black screen with GNU GRUB appears. Edit the line 

linux /boot/linux26 ro ramdisk_size=16777216 rw quiet splash=silent

so it reads 

linux /boot/linux26 ro ramdisk_size=16777216 rw nomodeset
  • By removing the quiet splash=silent we force the installer to show every single line of code as it loads. If it freezes again, we see the exact driver name where it stopped (e.g., sata_nv or i915).
  • Adding nomodeset: This is the "magic bullet" for the nvidiafb hang. It tells the kernel not to touch the video drivers until the system is fully loaded. It is most likely where the installer hangs. As we should see every line of the execution of the installer if we have a different driver issue we should still see it.

To save the changes press ctrl+x or maybe F10. The GNU Grub screen will close and the graphical installer will continue until it gets to the "end user license agreement (EULA) screen. For anyone else reading this, If the installer freezes again before it gets to the EULA screen the error will most likely be in the last laine of text displayed. The most likely error will be a conflict between Debian kernel and Advanced Configuration and Power Interface (ACPI) in which case restart and press the "e" to get to the Grub config and edit the same line 

linux /boot/linux26 ro ramdisk_size=16777216 rw nomodeset

so that it now has noapic added. note that this is just for the installer not the Proxmox application. 

linux /boot/linux26 ro ramdisk_size=16777216 rw nomodeset noapic

After modifying that line press ctrl+x or F10 to save and proceed. if it stalls again you will need to look for the exact error in the last line or last few lines that the installer performed or any line that starts or ends with failed.

Proxmox installer continued

After the earlier problems we should have got to the ELUA screen so press "I agree" button to continue. The next screen will be about choosing hard drives. Kiwi has a single SSD and three Ironwolf Pro 14 TB HDDs. Obviously, we will not install the OS onto a HDD or even all three HDDs as it would be too slow and as the SSD is fast we will choose that as the Target Harddisk. with the SSD selected click on options button.

  • select zfs (RAID0)
  • select samsung SSD as hardisk 0
  • select Do not use for harddisk 1
  • select Do not use for harddisk 2
  • select Do not use for harddisk 3

ZFS is the best choice as it has native compression, Bitrot detection but not really protection because there is only one drive so nowhere to restore any corrupt data from. ZFS also has good snapshot capability. Ext4 / XFS are "set it and forget it" options. They use almost zero RAM, but they offer no protection against data corruption and have very basic snapshotting capabilities (via LVM). Btrfs is similar to ZFS but generally considered less stable and has a lower feature set. The HDDs will not be used during the installation of Proxmox because it is easier to set them up from within the Proxmox web GUI or CLI.so the do not use is the best option for now note before we exit the Harddisk Options we need to check that the ashift is set to 12 so that the data blocks align with the physical page size of the Samsung 990 Pro's NAND, ensuring maximum speed, so select Advanced Options button next to Disk setup. Choose

  • ashift is 12
  • compress is on
  • checksum is on
  • copies is 1. Since this is a single-disk ZFS RAID0, copies=1 is the standard. If it is set it to 2, it would store every block twice on the same physical SSD. While that would allow ZFS to repair bitrot, it would effectively cut the 2TB SSD down to 1TB and halve the write speeds. With a 990 Pro, standard checksums and a good backup strategy are much more efficient.
  • ARC max size 6422 Since there are only have two RAM slots and we are capped at 64GB, we need to be very intentional about how we carve up Kiwi RAM. having said that it is only a Maximum not a set size and zfs does not have to use all of it's maximum
  • hdsize is set to maximum ( the + button will only allow it to be the size of the harddisk so click it until it stops going up)

Check the settings and when all looks correct click next for the regional settings

  • Country United Kingdom
  • Timezone Europe/London (or UTC if preferred)
  • Keyboard Layout United Kingdom

Then next to set password and email address screen. Set a strong password that you can remember for the administration of the host. If the password is lost it will be almost impossible to recover it so do not forget it. Also it will be the hackers dream to get hold of the Proxmox host's password in that all of the rest of the security is mainly built on top of the administration of the host. If a host in a cluster is compromised it is quite likely that the whole cluster is compromised. The email address is so Proxmox can send promotional or updates but in general they don't flood their users with spam so it is no problem adding in a real email address. When done Click next to se the Management Network configuration screen.

The Management Network Configuration will determine where the WebGUI will be presented from. Kiwi has two NICs, one is the 2.5 GB p/s wired NIC and the other is the wireless NIC. by default it is likely to be the wired NIC that is going to be the one used for the management interface, mainly because at this stage WiFi has not been set up yet. Set up the options as follows

  • Management Interface nic0 (the r8169 is the driver that Debian uses for wired nics) It is the one that is up and the Wifi nic is down so it is straight forward to establish which is which.
  • hostname will be kiwi.seaoffate.net it is safe for us to use this domain name because we have registered it. If the reader has no domain name it could be kiwi.local but it would be unwise to use something that is already register on the public Internet, it may appear to work but there will be odd unforeseen problems like getting updates or push notifications or some other odd thing that don't work because the reply is being sent to a different IP address.
  • IP address will be 192.168.1.112/24
  • Gateway will be 192.168.1.1
  • DNS will be 192.168.1.1

When this is done click next for the Summary. Make sure it is all correct and the auto reboot after successful installation then click install.

Post Installation

When the host finishes the install it will restart. It may show the Proxmox Intallation again if the USB is still plugged in depending on the boot options, if so remove the USB and restart the computer. once the computer has finished starting it should show a simple login screen with the web address that is the main administration panel. from this point on the screen, mouse and keyboard can and should be removed as they will probably be never be used again. Now Kiwi will be managed from a web browser. The address will be whatever was set in the Management Network Configuration section with the port number 8006. in the case of Kiwi it is http://192.168.1.112:8006 the browser will give a security warning as no SSL cert has been setup for it yet so click advanced or ignore or whatever the browser needs to get to the web address. Then there will be a bank Proxmox screen with a popup in the middle with the title Proxmox VE Login the login details are

  • Username root
  • Password your_password ( whatever was set in the password and email section of the instalation)
  • Realm is Linux PAM standard authentication
  • Language English - English

note username and password are case sensitive

Samsung 990 Pro series SSDs Firmware

Now we have logged in we need to check that the Samsung SSD has got up to date firmware, Note This is for Samsung 990 Pro SSDs and not all SSDs it is not even all Samsung SSDs just this particular model in the early versions. Select Kiwi from the left pane then towards the top right of the main pane there is a row of buttons reboot, shutdown, shell, bulk actions, and help click on shell. A new window will appear with root@kiwi already logged in. Type the following into the window 

 smartctl -a /dev/nvme0n1

look through the information for the SSD, under === START OF INFORMATION SECTION === it should have a firmware Version line. Make sure the version is not "1B2QJXD7" it should be safe if it starts with 3 or more. If the Samsung 990 Pro SSD does have the unsafe version stop now and update the firmware immediately or it is likely that it will stop working after a short time. The version on this host is Firmware Version: 5B2QJXD7

Fix the No Subscription nag and do system update

By default, Proxmox tries to use a paid repository that you can't access. We need to disable that and add the free community one. This assumes that it is Proxmox version 9.x. Proxmox 9 uses .sources files. We must disable the default enterprise links and add the community repository. Run these commands in the Kiwi shell 

# 1. Disable Enterprise PVE Repo
mv /etc/apt/sources.list.d/pve-enterprise.sources /etc/apt/sources.list.d/pve-enterprise.sources.bak
# 2. Disable Enterprise Ceph Repo
mv /etc/apt/sources.list.d/ceph.sources /etc/apt/sources.list.d/ceph.sources.bak
# 3. Create the No-Subscription Source
cat <<EOF > /etc/apt/sources.list.d/pve-no-subscription.sources
Types: deb
URIs: http://download.proxmox.com/debian/pve
Suites: trixie
Components: pve-no-subscription
Architectures: amd64
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF

The "No-Nag" UI Patch: This command neutralizes the subscription check in the web interface and restarts the proxy service to apply it 

# 4. Remove 'No Valid Subscription' Login Popup
sed -Ezi.bak "s/(Ext.Msg.show\(\{\s+title: gettext\(\'No valid sub)/void\(\{ \/\/\1/g" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js && systemctl restart pveproxy.service

When the system reboots open a new console window for the host. So we can do an update. Always use dist-upgrade for Proxmox to ensure kernel and dependency changes are handled correctly. 

# 5. Refresh and Upgrade
apt update && apt dist-upgrade -y

Summary of why we did this

  • The .bak method: Instead of fighting sed patterns that might change, moving the files to .bak is a "bulletproof" way to ensure apt ignores them.
  • Trixie Suites: Using Suites: trixie ensures you aren't pulling in old bookworm (Proxmox 8) packages that could destabilize your new Ryzen 5 build.
  • The 63 Packages: Now that you've run the update, those 63 packages are waiting. Running the dist-upgrade command above will install the latest 2026 security patches and hardware drivers.
Ironwolf Pro Harddrive Setup

Since we have already performed the dist-upgrade, the kernel is fresh and ready to handle the heavy I/O of those 14TB drives. We will set this up via the CLI to ensure we get the exact optimizations needed for a high-density vault. First, we must be 100% certain which device names (/dev/sdX) belong to the IronWolfs. 

lsblk -o NAME,SIZE,MODEL,SERIAL

Look for the three drives that show approximately 12.7T (the TiB equivalent of 14TB). Verify the Model says ST14000.... In this case they are

  • sda 12.7T ST14000NE0008-2JK101 ZTM0D5FK
  • sdb 12.7T ST14000NE0008-2H4101 ZL2EJCXX (this one has a partition sdb1 but we ignore it)
  • sdc 12.7T ST14000NE0008-2JK101 ZTM0D5A0

The Clean Slate (Wipe the Labels) Run these commands to clear the old ZFS and EFI labels. This doesn't delete all of the old data (which takes hours), it just "un-formats" the start and end of the disks so ZFS sees them as new. It may not be necessary for all of the drives but certainly is for sdb as that has a partition on it 

# Wipe the ZFS labels specifically
zpool labelclear -f /dev/sda
zpool labelclear -f /dev/sdb
zpool labelclear -f /dev/sdc
# Wipe the partition tables entirely (the "Nuclear" option)
wipefs -a /dev/sda
wipefs -a /dev/sdb
wipefs -a /dev/sdc

The other drive is the 990 Pro SSD. Now we can create the ZFS Z1 Pool We will name the pool kiwipool. We are using ashift=12 for 4K sector alignment and atime=off to save the mechanical heads from unnecessary movement. 

zpool create -o ashift=12 kiwipool raidz1 /dev/sda /dev/sdb /dev/sdc
# Apply optimizations
zfs set compression=lz4 kiwipool 
zfs set atime=off kiwipool 
zfs set xattr=sa kiwipool 
zfs set recordsize=1M kiwipool

once that is done we can verify the pool is built with 

zpool status kiwipool

Next we need to make some datasets to keep the pool organised and not write anything to the root of kiwipool. Run thes commands 

zfs create kiwipool/PFK
zfs create kiwipool/PFK/pdata
zfs create kiwipool/PFK/pdata/KiwiHDs
zfs create kiwipool/ISO
zfs create kiwipool/PFK/backups
zfs list

The zpool will show up in the webgui in Kiwi->storage and kiwi->zfs but not in the VM list on the left hand pane so they cannot be added to VMs yet. To make them available we run the command 

pvesm add zfspool KiwiHDs --pool kiwipool/PFK/pdata/KiwiHDs --content rootdir,images
pvesm add dir KiwiISOs --path /kiwipool/ISO --content iso,vztmpl
pvesm add dir KiwiBackups --path /kiwipool/PFK/backups --content backup

Now in the webgui there should be the drives for VM HDs, ISOs and backups

Check and configure the WiFi

Testing Wi-Fi on a Proxmox (Debian) host is a bit different than on a laptop because there is no "taskbar" to click. We have to do this through the CLI. First, let's see if Debian even sees the Wi-Fi card and has the drivers loaded. 

ip link

Look for: An interface starting with w (e.g., wlp2s0 or wlan0). If you don't see one the drivers might be missing or it's "Hard Blocked" in the BIOS. In this case The hardware check is a success. Your Wi-Fi card is identified as wlp5s0 (it did show up in the installer but it was ignored then ). The state is currently DOWN, which is expected since it hasn't been configured with your SSID or credentials yet. Because Proxmox is built for servers, it lacks a graphical Wi-Fi picker. We have to manually tell the background service (wpa_supplicant) how to talk to your router. at the same time install the wireless tools package. 

apt update && apt install wpasupplicant wireless-tools -y

Instead of typing the Wi-Fi password in plain text, we’ll use a tool to generate a secure configuration block. Replace Your_SSID and Your_Password with your actual wireless AP details: 

wpa_passphrase "Your_SSID" "Your_Password" > /etc/wpa_supplicant/wpa_supplicant.conf

Now we can test the connection let's see if Kiwi can actually authenticate. 

wpa_supplicant -B -i wlp5s0 -c /etc/wpa_supplicant/wpa_supplicant.conf

Then wait a few seconds before checking the status with 

iw wlp5s0 link

Assuming it shows a connection we can make it permanent. To ensure the Wi-Fi comes up automatically on boot and acts as a secondary management port, we need to add it to the network interfaces file. Warning: Be very careful with the syntax here. Use this command to append the configuration to /etc/network/interfaces. cat <<EOF >> /etc/network/interfaces 

auto wlp5s0
iface wlp5s0 inet static
        address 192.168.1.113/24
        wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf
EOF
🌐 Networking & The "Wi-Fi Failover" Battle

This section documents the attempt to create a reliable wireless "emergency backdoor." The objective was to have the Wi-Fi (192.168.1.113) act as a secondary management port that remains accessible even if the 2.5G cable is pulled. The Problem was cuased by (Asymmetric Routing). Linux prefers the route with the lowest metric. Since the 2.5G wire has a metric of 10 and Wi-Fi has 100, pulling the cable caused a "Routing Black Hole"—Kiwi would receive requests via Wi-Fi but try to reply via the dead Ethernet cable.

Several fixes were applied including

  • Kernel Routing Tweaks: Added to /etc/sysctl.conf to force the OS to ignore dead routes immediately
nano /etc/sysctl.conf

net.ipv4.conf.all.ignore_routes_with_linkdown=1
net.ipv4.conf.default.ignore_routes_with_linkdown=1
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2
  • We tried to create service persistence. We created a systemd override for wpa_supplicant@wlp5s0 to handle driver timing issues
[Service]
Restart=on-failure
RestartSec=5s
ExecStartPost=/usr/sbin/ifup wlp5s0 --force
Retrieved from "https://wiki.seaoffate.net/index.php?title=Proxmox_Server&oldid=1512"