Network Configuration

Introduction

With the addition of a second host, Kiwi, some new decisions need to be made on how to access its services. At the same time now is a good time to increase the throughput in the entire Home Lab

Hardware Configuration

Some new hardware has been added as follows.

🍐Pear

Pear is the original host and was setup with a single 1 Gb p/s NIC. As it only had one NIC the management and LAN were both using the it. Now we have added a second 2.5 Gb p/s NIC. The management is still via the original NIC and the WAN port of Pfsense is also still on the original NIC

• Management port is 192.168.1.111:8006
• Pfsense WAN Port is 192.168.1.125

🥝Kiwi

Kiwi is the new host with a single 2.5 GB p/s NIC and a WIFI NIC. However the Wifi only functions as a management connection if the wired NIC is doing the same so wifi is not particularly useful at this time. WiFi does have an IP address for the management port as 192.168.1.113 as it does technically function. Because we will need to keep the cabled connection to the management port up, we will need to set the Managed switch to untagged for vlan1 on Kiwi's port (port2). The side effect of having VLAN1 as untagged is that if any VMs or LXCs are not set to a VLAN they will have direct access to the internet through VLAN1/vmbr0/nic0.

• Management port (2.5GbP/s NIC) is 192.168.1.112:8006
• Management port (Wifi) is 192.168.1.113:8006

🍇Grape

The desktop PC Grape has also now got a new 2.5 GB p/s NIC that will be connected to the new managed switch and the VLANs some caution will be needed as the original NIC connects to the internet on 192.168.1.1/24 so some care needs to be taken to stop any of the home lab VLANs from bypassing the Pfsense firewall and linking directly to the internet.

🔀Switches

There is the original Dumb 4 port switch connected to ISP, Desktop PC, Pear's 1 Gb p/s NIC, and port 1 of the new managed switch. Now there is new managed 4 +2 switch (the 2 SFPs lots are unused).

Initial considerations

Before we start we need to do some checking and configuration to make sure the NICs don't change names and fail to load drivers thereby locking us out of the Webgui.

Identify and Pin the Onboard NIC

Update The following can be done but it was more trouble than it was worth. NICs usually take on the names of the PCI slots that they use and can be identified in the console logon. the only snag is that we would need to actually use a keyboard and screen if the NIC changes name.

Before adding any new hardware, you can "lock" the onboard NIC so its name never changes, even if the PCI bus reorders itself. Find the MAC Address: In the Shell, either or look at System > Network in the webgui or run

ip link

Find the onboard NIC (likely enp6s0 originally) and copy its MAC address (e.g., aa:bb:cc:dd:ee:ff). Use the MAC address to create the Udev Rule

nano /etc/udev/rules.d/70-persistent-net.rules

and add the line (substitute aa:bb:cc:dd:ee:ff for the actual MAC address)

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="aa:bb:cc:dd:ee:ff", NAME="enonboard0"

Next we need to update the network config

nano /etc/network/interfaces

find the old interface name (probably enp6s0) and replace it with

enonboard0

Installing the 2.5G Driver

The original driver for the 2.5 NIC is buggy and doesn't work properly and needs to be replaced. Update The later versions of pve have an updated the native r8169 driver so it work on realtek 2.5 NIC. so we can skip the next step and not bother with the 3rd party r8125 driver

apt update
apt install -y pve-headers-$(uname -r)
apt install -y r8125-dkms

then blacklist the old Driver

echo "blacklist r8169" > /etc/modprobe.d/blacklist-r8169.conf

To force Load the New Driver

echo "r8125" >> /etc/modules

Updating the Boot Process

We need to "bake" the new names and drivers into the kernel's startup memory to prevent "Phantom" NICs. First refresh Initramfs: This updates the initial RAM disk that the kernel uses during the first seconds of boot. Please note this may take a while to complete. Update if we don't install the r8125 driver we dont need to do this either.

update-initramfs -u

Refresh Boot Tool: This ensures the changes are synced to the EFI partition.

proxmox-boot-tool refresh

Now we can shutdown and physically install the 2.5 NIC

What to do if the NIC changes name

If the onboard NIC changes name when a second NIC is added the webgui will be inaccessible so time to dig out a keyboard and screen. The first step is to login to the console. Then we need to see what links are up but the problem will be that the console will show many NICs from all of the VMs so it is easier to have them in a text file like

ip link -br link > link.txt
nano link.txt

We need to look for the enpxxx line with LOWER_UP, assuming that the only NIC with Cable is the on board one. it should say something like

enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP>

if we can't find the lines that we want in the mass of VM NICs we could cut the list down with

ip -br link show type device

If that is no good we could check the hardware because one of the NICs is onboard and the other is on a PCIE slot. Onboard NICs are usually built into the chipset (Bus 00 or 06), while PCIe cards show up on a separate expansion bridge. The following command should show one line with enp4s0 for the PCI NIC and one line with enp6s0 for the onboard NIC

ls -l /sys/class/net | grep pci

If we are still not sure we can get the MAC address from a sticker on the NIC or more likely get it from BIOS and do

cat /sys/class/net/enp6s0/address

Once we have the correct name for the onboard NIC we can edit the file

/etc/network/interfaces

and replace the old enpxs0 name in vmbr0 with the new name. After saving the interfaces file we will need to test with the command

 ifreload -a

Or if we want to force an interface to come up we take it down then up

ifdown vmbr0 && ifup vmbr0

note if we take vmbr0 down and up while on a ssh or webgui it will break our own connection so only do so when directly logged on to a console. When we are sure that onboard NIC is working we need to make sure the change is permanent. So remove any pending GUI changes

rm /etc/network/interfaces.new

Sync the boot image:

update-initramfs -u

Sync the bootloader:

proxmox-boot-tool refresh

and finally reboot

Potential Problems

The problems that need to be avoided are mainly not allowing any path to internet that is not supposed to be allowed and not having any routing loops

VLANS

Old setup vs new VLANs

The old protected LANs on Pear were all simply unbridged networks that is networks without any physical NIC so only existed within Proxmox. The new network setup deletes all of those floating networks or networks with no bridge and has one network vmbr1 that is network aware and all of the VLANs are within this single Network.

• Name vmbr1
• Type is network bridge
• autostart ticked
• VLAN Aware ticked
• Bridge ports is enp4s0
• MTU is 1500 (default setting)
• VLAN IDs are 2-4094 (default setting)

New VLAN Settings

• ISP Gateway: 192.168.1.1
• Management Subnet: 192.168.1.0/24 (Physical Dumb Switch)
• Production Backbone: 2.5Gbps Managed Switch (VLAN Tagged)

VLAN & Subnet Map

VLAN ID	Name	Subnet	Purpose
1	Management	192.168.1.0/24	Proxmox GUIs, pfSense WAN, ISP
100	Production	192.168.100.0/24	Webservers, MYSQL servers, gameservers any other services
110	Infrastructure	192.168.110.0/24	Internal services (DNS, NTP, Auth)
130	VPNNet	192.168.130.0/24	WireGuard / OpenVPN VMs
120	Lab	192.168.120.0/24	Sandboxed testing / POCs
99	MGT (Internal)	192.168.99.0/24	Internal server management (Pfsense, Prometheus and etc)
111	Terminal	192.168.111.0/24	RDP / NoMachine gateways

SDN configuration on Proxmox

There is no central list of VLANs with IDs and names on the Proxmox host as it uses the Software Defined Network, SDN, philosophy. Instead of having several networks in the system->networks section of the webGUI we have a single network (vmbr1) bridged to a single physical NIC (enp4s0) which is VLAN aware and each individual VM will have the bridge set to the one network (vmbr1) and have a VLAN tag set to whatever VLAN is required. So it looks a bit odd compared to a managed switch method of defining VLANs in a table. If a listing of the VLANs is required

bridge vlan show

will list all 4094 vlans.

Physical Switch Configuration

Dumb Switch (1Gbps Management) Connects

• ISP Router
• Pear 1Gbps NIC
• Grape (Win11 Desktop PC's 1Gbps NIC)
• Managed Switch Port 1.

Managed Switch Configuration

Port	Host	VLAN1	VLAN100	VLAN 99,110,120,130
1	🐌dumb switch	untagged	N/C	N/C
2	🥝Kiwi	Untagged	Tagged	Tagged
3	🍇Grape	N/C	Untagged	N/C
4	🍐Pear	N/C	Tagged	Tagged

Testing

We can prove that the VLANs work by changing the tag on a webserver and refreshing the page, if it works with the tag but fails without it we can assume that Proxmox is recognising the tags. Another obvious test would be logon to a vm from the console and ping some other network on different subnet and some on the same subnet, again with and without the correct tag, rember that Pfsense may permit ping between subnets so switch it off before testing.