Network Configuration

Introduction

With the addition of a second host, Kiwi, some new decisions need to be made on how to access its services. At the same time now is a good time to increase the throughput in the entire Home Lab

Hardware Configuration

Some new hardware has been added as follows.

Pear

Pear is the original host and was setup with a single 1 Gb p/s NIC. As it only had one NIC the management and LAN were both using the it. Now I have added a second 2.5 Gb p/s NIC. The management is still via the original NIC and the WAN port of Pfsense is also still on the original NIC

• Management port is 192.168.1.111:8006
• Pfsense WAN Port is 192.168.1.125

Kiwi

Kiwi is the new host with a single 2.5 GB p/s NIC and a WIFI NIC. It will have no direct access to the Internet and the WIFI NIC will probably be disabled. The single 2.5 NIC will have to be used as the management port. The VLANs will also be connected to this 2.5 NIC and hence through to Pear and it's Pfsense gateways.

• Management port is 192.168.1.112:8006

Grape

The desktop PC Grape has also now got a new 2.5 GB p/s NIC that will be connected to the new managed switch and the VLANs some caution will be needed as the original NIC connects to the internet on 192.168.1.1/24 so some care needs to be taken to stop any of the home lab VLANs from bypassing the Pfsense firewall and linking directly to the internet.

Switches

There is the original Dumb 4 port switch connected to ISP, Desktop PC, Pear's 1 Gb p/s NIC, and port 1 of the new managed switch. Now there is new managed 4 +2 switch (the 2 SFPs lots are unused).

Initial considerations

Before we start we need to do some checking and configuration to make sure the NICs don't change names and fail to load drivers thereby locking us out of the Webgui

Identify and Pin the Onboard NIC

Before adding any new hardware, you must "lock" the onboard NIC so its name never changes, even if the PCI bus reorders itself. Find the MAC Address: In the Shell, either or look at System > Network in the webgui or run

ip link

Find the onboard NIC (likely enp6s0 originally) and copy its MAC address (e.g., aa:bb:cc:dd:ee:ff). Use the MAC address to create the Udev Rule

nano /etc/udev/rules.d/70-persistent-net.rules

and add the line (substitute aa:bb:cc:dd:ee:ff for the actual MAC address)

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="aa:bb:cc:dd:ee:ff", NAME="enonboard0"

Next we need to update the network config

nano /etc/network/interfaces

find the old interface name (probably enp6s0) and replace it with

enonboard0 

Installing the 2.5G Driver

The original driver for the 2.5 NIC is buggy and doesn't work properly and needs to be replaced.

apt update
apt install -y pve-headers-$(uname -r)
apt install -y r8125-dkms

then blacklist the old Driver

echo "blacklist r8169" > /etc/modprobe.d/blacklist-r8169.conf

To force Load the New Driver

echo "r8125" >> /etc/modules

Updating the Boot Process

We need to "bake" the new names and drivers into the kernel's startup memory to prevent "Phantom" NICs. First refresh Initramfs: This updates the initial RAM disk that the kernel uses during the first seconds of boot. Please note this may take a while to complete.

update-initramfs -u

Refresh Boot Tool: This ensures the changes are synced to the EFI partition.

proxmox-boot-tool refresh

Now we can shutdown and physically install the 2.5 NIC

Potential Problems

The problems that need to be avoided are mainly not allowing any path to internet that is not supposed to be allowed and not having any routing loops

VLANS

Old setup vs new VLANs

The old protected LANs on Pear were all simply unbridged networks that is networks without any physical NIC so only exist within Proxmox. The new network setup deletes all of those floating networks or networks with no bridge and has one network vmbr1 that is network aware and all of the VLANs are within this single Network.

• Name vmbr1
• Type is network bridge
• autostart ticked
• VLAN Aware ticked
• Bridge ports is enp4s0
• MTU is 1500 (default setting)
• VLAN IDs are 2-4094 (default setting)

New VLAN Settings

• ISP Gateway: 192.168.1.1
• Management Subnet: 192.168.1.0/24 (Physical Dumb Switch)
• Production Backbone: 2.5Gbps Managed Switch (VLAN Tagged)

VLAN & Subnet Map

VLAN ID	Name	Subnet	Purpose
1	Management	192.168.1.0/24	Proxmox GUIs, pfSense WAN, ISP
100	Production	192.168.100.0/24	Webservers, MYSQL servers, gameservers any other services
110	Infrastructure	192.168.110.0/24	Internal services (DNS, NTP, Auth)
130	VPNNet	192.168.130.0/24	WireGuard / OpenVPN VMs
120	Lab	192.168.120.0/24	Sandboxed testing / POCs
99	MGT (Internal)	192.168.99.0/24	Internal server management (Pfsense, Prometheus and etc)
111	Terminal	192.168.111.0/24	RDP / NoMachine gateways

SDN configuration on Proxmox

There is no central list of VLANs with IDs and names on the Proxmox host as it uses the Software Defined Network, SDN, philosophy. Instead of having several networks in the system->networks section of the webGUI we have a single network (vmbr1) bridged to a single physical NIC (enp4s0) which is VLAN aware and each individual VM will have the bridge set to the one network (vmbr1) and have a VLAN tag set to whatever VLAN is required. So it looks a bit odd compared to a managed switch method of defining VLANs in a table. If a listing of the VLANs is required

bridge vlan show

will list all 4094 vlans.

Physical Switch Configuration

Dumb Switch (1Gbps Management) Connects ISP Router, Pear 1Gbps NIC, Win11 Desktop PC's 1Gbps NIC, and Managed Switch Port 1.

Managed Switch (2.5Gbps Backbone)

• Port 1 (Uplink to Dumb Switch):
  • VLAN 1: Untagged (PVID 1)
  • All other VLANs: Excluded (Prevents leaks to ISP router)

• Port 4 (Pear - Host 1):
  • VLAN 1: Excluded (Management stays on 1Gbps NIC)
  • VLANs 30, 110, 130, 120, 99, 111: Tagged

• Port 2 (Kiwi - Host 2):
  • VLAN 1: Tagged (Enables management over single NIC)
  • VLANs 30, 110, 130, 120, 99, 111: Tagged

• Port 3 (Win11 Desktop):
  • VLAN 1: Untagged (Optional fallback)
  • VLAN 30: Tagged (For 2.5G production access)

Testing

We can prove that the VLANs work by changing the tag on a webserver and refreshing the page, if it works with the tag but fails without it we can assume that Proxmox is recognising the tags. Another obvious test would be logon to a vm from the console and ping some other network on different subnet and some on the same subnet, again with and without the correct tag, rember that Pfsense may permit ping between subnets so switch it off before testing.