Network Configuration

Introduction

With the addition of a second host, Kiwi, some new decisions need to be made on how to access its services. At the same time now is a good time to increase the throughput in the entire Home Lab

Problems & Solutions

As any problems arise they will need to be recorded so that if the same problem occurs in the future we should have a fix in place.

Diagram

A Dagram of the entire network can be seen here

The diagram of the network. (click to expand) .

we need to edit the above to include

Kiwiberry 192.168.100.40 on kiwi
change the hostname for mgtcinnamon to Cinnamon
at the moment lychee does not exist so will need to be re built
Add Mango 192.168.110.133 as a replacement for the Prometheus, Grafana Victoria Metrics trio
add kapok 192.168.100.45

Hardware Configuration

Some new hardware has been added as follows.

🍐Pear

Pear is the original host and was setup with a single 1 Gb p/s NIC. As it only had one NIC the management and LAN were both using the it. Now we have added a second 2.5 Gb p/s NIC. The management is still via the original NIC and the WAN port of Pfsense is also still on the original NIC.

Origins of Pear

When Pear was setup at the beginning it was the only Proxmox host and there was no managed switch. There was no need to do VLANs as there was nothing to connect to. The Pfsence VM had a vmbr0 as the WAN port with an IP address 192.168.0.125 on the same broadcast domain as the ISP router LAN side and as the router had few options for port forwarding everything was forwarded to Pfsense. The Pfsense VM had 6 other virtual NICs that had no physical NIC or vmbr attached so they floated in the air with no connection to the outside world, so each of the networks were completely isolated from the other networks and the physical world. So every VM would have its own virtual NIC with a gateway address on the Pfsense firewall. With the floating networks isolated it was no problem having the Proxmox management port sharing the connection to the WAN port of Pfsense as long as any attempt to reach 192.168.0.0/24 was blocked by Pfsense and whatever happened to the VMs on the LAN side of Pfsense they could never access the management port of Pear. The only problem with this setup was that there is no way for those LAN side host to connect with any physical device on any other server other than through Pfsense so with the arrival of the new Proxmox host Kiwi the only possible method to connect the two hosts together would be to install Pfsense on Kiwi and set up IPSEC between them but that is quite a heavy duty protocol for a small homelab. Added to this mixture is the point that Kiwi has a 2.5 GBp/s NIC and a Wifi that was potentially not going to work. So it was decided to get a managed switch with 2.5 GBp/s capability. Having decided that we would like to increase speed between the two hosts a new NIC was bought at the same time and a second NIC for Grape to enable it to share in the higher speed transfers especially as the cards were only £13.26 each and the managed switch was less than £30. I suppose I could have got 10GBp/s NICs at the time and either 10GBp/s STPs or a fast switch but it was decided not to bother at this time as Kiwi will never have a different NIC because there are no spare PCIe slots on the ITX MB. When the managed switch is plugged in to Pear the switch port will not be connected to VLAN1

New VLANs for Pear

Having decided to invest a grand total of about £60 to get the transfer speeds up to 2.5 times what I had we had to give some thought to the layout of the network because we could not continue with the the Isolated Bridge networks anymore with more than one host. It has been decided to keep the onboard NIC connected to the management interface and the WAN port of Pfsense. The Isolated Bridge networks needed to be changed to VLANs, instead of six separate networks floating in RAM we now have a single network vmbr1 that is slaved to the new 2.5GBp/s NIC with 6 VLANs. Unlike the previous Isolated Bridge networks setup that was defined in the network section of the WebGUI VLANs are set on the host VM's hardware section on the NIC, it is simply a case of assigning the network bridge and then adding the VLAN number. The definition of vmbr1 is a bit different in that we do not want to connect the management Interface of PVE from this network so the easiest way to stop any VM from accessing PVE is to not have any IP address assigned to vmbr1. We do need it to be VLAN aware so we need to enable that function and we do need it to auto start. It should also be listening to VLAN IDs 2-4096. When the managed switch is plugged in to Pear the switch port will not be connected to VLAN1 and will be Tagged to all of the other VLANs. Each Virtual Machine on Pear will be on one of the VLANs if it not the traffic will go nowhere for any that are missing because there is IP address on the vmbr1 no traffic will be the default

Management port is 192.168.1.111:8006
Pfsense WAN Port is 192.168.1.125:8006

🥝Kiwi

Kiwi is the new host with a single 2.5 GB p/s NIC and a WIFI NIC. However the Wifi only functions as a management connection if the wired NIC is doing the same so wifi is not particularly useful at this time. WiFi does have an IP address for the management port as 192.168.1.113 as it does technically function. Because we will need to keep the cabled connection to the management port up, we will need to set the Managed switch to untagged for vlan1 on Kiwi's port (port2). The side effect of having VLAN1 as untagged is that if any VMs or LXCs are not set to a VLAN they will have direct access to the internet through VLAN1/vmbr0/nic0.

Management port (2.5GbP/s NIC) is 192.168.1.112:8006
Management port (Wifi) is 192.168.1.113:8006

🍇Grape

The desktop PC Grape has also now got a new 2.5 GB p/s NIC that will be connected to the new managed switch and the VLANs some caution will be needed as the original NIC connects to the internet on 192.168.1.1/24 so some care needs to be taken to stop any of the home lab VLANs from bypassing the Pfsense firewall and linking directly to the internet. At present the HyperV virtualisation on Grape has only been setup ant tested but there are no networks or guests defined within the HyperV environment. The 2.5 NIC on Grap has been renamed to Fruitnet to make it easier to identify. Fruitnet now has a fixed IP address of 192.168.100.50, DNS server is set to 192.168.110.11 with seaoffate.net and seaoffate.local search domains the metric for the interface is now 100 and there is no gateway set. The other onboard NIC only needed to have the interface metric change to 5. There was some difficulty finding the network settings as it has moved out of the system tray and into Control Panel\Network and Internet\Network and Sharing Center. Some time ago a few entries were made in the hosts file so that all of the webservers would have IP address of Pfsense WAN port these have all been removed now so that the DNS at 192.168.110.11 will serve the local IP addresses through the fruitnet NIC. The next thing to configure is the routes.

The Fruitnet NIC is untagged on VLAN100 (Production) and has an IP address 192.168.100.50/24 so can directly connect to any of the hosts there but the connection to any other of the networks would need to go through Pfsense on 192.168.100.1/24 and for all of the other hosts that work ok as that is there gateway. However, Grape on fruitnet has no gateway set so that all internet access will have to go via the onboard NIC and the ISP, which works fine for everything except the internal traffic because Grape & ISP don't know how to connect to any internal network. The solution is to add some static routes to the other networks that set Pfsense as the forwarding IP address. The following commands should be entered into an elevated Powershell console.

route -p add 192.168.110.0 mask 255.255.255.0 192.168.100.1 IF 12
route -p add 192.168.111.0 mask 255.255.255.0 192.168.100.1 IF 12
route -p add 192.168.120.0 mask 255.255.255.0 192.168.100.1 IF 12
route -p add 192.168.130.0 mask 255.255.255.0 192.168.100.1 IF 12

now there is route to all of the networks as defined on Pear and Kiwi. To check the list of routes set on grape do

route print -4

It should be possible to set a single route to 192.168.0.0/16 but that would rely on the metrics being correct both now and in the future and this explicit set of routes will set all of the routes as a metric of 1 so Fruitnet/Pfsense will always have the highest priority route through to these networks.

To summarise what we have done on Grape

set the name of the 2.5GBp/s Network Interface Card to Fruitnet
Set the IP address of Fruitnet to 192.168.100.50/24 with a metric of 5 so that it always gets priority. Set DNS to the internal DNSMasq 192.1638.111.11 and set the two search domains. finally left the gateway as blank
Set the metric of the onboard NIC to 10 so that it loses priority to the Fruitnet NIC but is still the failover
Set the VLAN 100 as untagged so that all 192.168.100.0/24 traffic will be passed on by the managed switch.
Set the routes to all other networks to be passed on by Pfsense 192.168.100.1

🔀Switches

There is the original Dumb 4 port switch connected to ISP, Desktop PC, Pear's 1 Gb p/s NIC, and port 1 of the new managed switch. Now there is new managed 4 +2 switch (the 2 SFPs lots are unused).

Initial considerations

Before we start we need to do some checking and configuration to make sure the NICs don't change names and fail to load drivers thereby locking us out of the Webgui.

Identify and Pin the Onboard NIC

Update The following can be done but it was more trouble than it was worth. NICs usually take on the names of the PCI slots that they use and can be identified in the console logon. the only snag is that we would need to actually use a keyboard and screen if the NIC changes name.

Before adding any new hardware, you can "lock" the onboard NIC so its name never changes, even if the PCI bus reorders itself. Find the MAC Address: In the Shell, either or look at System > Network in the webgui or run

ip link

Find the onboard NIC (likely enp6s0 originally) and copy its MAC address (e.g., aa:bb:cc:dd:ee:ff). Use the MAC address to create the Udev Rule

nano /etc/udev/rules.d/70-persistent-net.rules

and add the line (substitute aa:bb:cc:dd:ee:ff for the actual MAC address)

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="aa:bb:cc:dd:ee:ff", NAME="enonboard0"

Next we need to update the network config

nano /etc/network/interfaces

find the old interface name (probably enp6s0) and replace it with

enonboard0

Installing the 2.5G Driver

The original driver for the 2.5 NIC is buggy and doesn't work properly and needs to be replaced. Update The later versions of pve have an updated the native r8169 driver so it work on realtek 2.5 NIC. so we can skip the next step and not bother with the 3rd party r8125 driver

apt update
apt install -y pve-headers-$(uname -r)
apt install -y r8125-dkms

then blacklist the old Driver

echo "blacklist r8169" > /etc/modprobe.d/blacklist-r8169.conf

To force Load the New Driver

echo "r8125" >> /etc/modules

Updating the Boot Process

We need to "bake" the new names and drivers into the kernel's startup memory to prevent "Phantom" NICs. First refresh Initramfs: This updates the initial RAM disk that the kernel uses during the first seconds of boot. Please note this may take a while to complete. Update if we don't install the r8125 driver we dont need to do this either.

update-initramfs -u

Refresh Boot Tool: This ensures the changes are synced to the EFI partition.

proxmox-boot-tool refresh

Now we can shutdown and physically install the 2.5 NIC

What to do if the NIC changes name

If the onboard NIC changes name when a second NIC is added the webgui will be inaccessible so time to dig out a keyboard and screen. The first step is to login to the console. Then we need to see what links are up but the problem will be that the console will show many NICs from all of the VMs so it is easier to have them in a text file like

ip link -br link > link.txt
nano link.txt

We need to look for the enpxxx line with LOWER_UP, assuming that the only NIC with Cable is the on board one. it should say something like

enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP>

if we can't find the lines that we want in the mass of VM NICs we could cut the list down with

ip -br link show type device

If that is no good we could check the hardware because one of the NICs is onboard and the other is on a PCIE slot. Onboard NICs are usually built into the chipset (Bus 00 or 06), while PCIe cards show up on a separate expansion bridge. The following command should show one line with enp4s0 for the PCI NIC and one line with enp6s0 for the onboard NIC

ls -l /sys/class/net | grep pci

If we are still not sure we can get the MAC address from a sticker on the NIC or more likely get it from BIOS and do

cat /sys/class/net/enp6s0/address

Once we have the correct name for the onboard NIC we can edit the file

/etc/network/interfaces

and replace the old enpxs0 name in vmbr0 with the new name. After saving the interfaces file we will need to test with the command

 ifreload -a

Or if we want to force an interface to come up we take it down then up

ifdown vmbr0 && ifup vmbr0

note if we take vmbr0 down and up while on a ssh or webgui it will break our own connection so only do so when directly logged on to a console. When we are sure that onboard NIC is working we need to make sure the change is permanent. So remove any pending GUI changes

rm /etc/network/interfaces.new

Sync the boot image:

update-initramfs -u

Sync the bootloader:

proxmox-boot-tool refresh

and finally reboot

Potential Problems

The problems that need to be avoided are mainly not allowing any path to internet that is not supposed to be allowed and not having any routing loops

VLANS

Old setup vs new VLANs

The old protected LANs on Pear were all simply unbridged networks that is networks without any physical NIC so only existed within Proxmox. The new network setup deletes all of those floating networks or networks with no bridge and has one network vmbr1 that is network aware and all of the VLANs are within this single Network.

Name vmbr1
Type is network bridge
autostart ticked
VLAN Aware ticked
Bridge ports is enp4s0
MTU is 1500 (default setting)
VLAN IDs are 2-4094 (default setting)

New VLAN Settings

ISP Gateway: 192.168.1.1
Management Subnet: 192.168.1.0/24 (Physical Dumb Switch)
Production Backbone: 2.5Gbps Managed Switch (VLAN Tagged)

VLAN & Subnet Map
VLAN ID	Name	Subnet	Purpose
1	Management	192.168.1.0/24	Proxmox GUIs, pfSense WAN, ISP
100	Production	192.168.100.0/24	Webservers, MYSQL servers, gameservers any other services
110	Infrastructure	192.168.110.0/24	Internal services (DNS, NTP, Auth)
130	VPNNet	192.168.130.0/24	WireGuard / OpenVPN VMs
120	Lab	192.168.120.0/24	Sandboxed testing / POCs
99	MGT (Internal)	192.168.99.0/24	Internal server management (Pfsense, Prometheus and etc)
111	Terminal	192.168.111.0/24	RDP / NoMachine gateways

SDN configuration on Proxmox

There is no central list of VLANs with IDs and names on the Proxmox host as it uses the Software Defined Network, SDN, philosophy. Instead of having several networks in the system->networks section of the webGUI we have a single network (vmbr1) bridged to a single physical NIC (enp4s0) which is VLAN aware and each individual VM will have the bridge set to the one network (vmbr1) and have a VLAN tag set to whatever VLAN is required. So it looks a bit odd compared to a managed switch method of defining VLANs in a table. If a listing of the VLANs is required

bridge vlan show

will list all 4094 vlans.

Physical Switch Configuration

Dumb Switch (1Gbps Management) Connects

ISP Router
Pear 1Gbps NIC
Grape (Win11 Desktop PC's 1Gbps NIC)
Managed Switch Port 1.

Managed Switch Configuration
Port	Host	VLAN1	VLAN100	VLAN 99,110,120,130
1	🐌dumb switch	untagged	N/A	N/A
2	🥝Kiwi	Untagged	Tagged	Tagged
3	🍐Pear	N/A	Tagged	Tagged
4	🍇Grape	N/A	Untagged	N/A

Testing

We can prove that the VLANs work by changing the tag on a webserver and refreshing the page, if it works with the tag but fails without it we can assume that Proxmox is recognising the tags. Another obvious test would be logon to a vm from the console and ping some other network on different subnet and some on the same subnet, again with and without the correct tag, remember that Pfsense may permit ping between subnets so switch it off before testing.