View source for Kiwi & OpenAlex failure while indexing the 2026 Import

==Introduction==

The problem with the indexing of the '''[[OpenAlex]]''' files was one of the first '''[[Network Configuration | network related]]''' problems with the '''[[Home Lab]]'''. Since the "I/O Storm" and the resulting network blackout were complex, it is will be useful to document the sequence of events.

==The Initial Problem==

===First Symptoms===

The first sign was that some of the SSH sessions to tayberry, the VM hosting docker and the OpenAlex container, closed unexpectedly and some became unresponsive so had to be forced to close. The management  interface for Kiwi also would not open in a web browser, the final thing was that the console for Proxmox only displayed a blinking cursor. All of the management for Kiwi was unresponsive so to shutdown kiwi we did the R E I S U B reboot so as to not corrupt the ZFS drives. To do the shutdown without pulling the plug we used the physical keyboard attached to Kiwi and held down Alt & PrintScreen and typed R E I S U B  (Raw keyboard, End tasks, Interrupt tasks, Sync disks, Unmount, Boot). When the machine rebooted it had the insert a bootable drive screen so probably just a bios confusion thing where the boot order got scrambled, a cold start fixed it and it booted to the normal login.

===Root Cause I/O Storm" & ARC Exhaustion ===

The problem was ZFS Scrub was initiated on the 14TB mechanical drives while Tayberry was performing heavy Random Write indexing into a 1.9TB OpenSearch index. The ZFS ARC (cache) was unconstrained, expanding to consume nearly all Host RAM. This caused a "Kernel Stall" on Kiwi, leading to a hard lockup of the Proxmox management interface and all guest VMs(all VMs was actually just one).

===The Solution to the Initial Problem===

First was the hard reset using R E I S U B forced power cycle of the Kiwi host and a cold boot reset the boot option back to SSD. To stop the problem from happening again we needed to limit the ZFS ARC to no more than 16gb RAM to ensure the Host OS and VM RAM remain protected during high I/O events. It is most likely that we will cut ZFS ARC down to 8 or 10GB as the system has only 64gb total. more information ac be found on the '''[[ZFS Commands]]''' pages. The quick way to set the maximum ARC size is to copy the value directly in to the module parameters but it will only be in effect until a reboot using echo command like:-
 echo 17179869184 > /sys/module/zfs/parameters/zfs_arc_max
But it would have to be written to /etc/modprobe.d/zfs.conf and the initramfs would need to be updated. see the '''[[ZFS Commands#ARC: Understanding the RAM usage | ARC: Understanding the RAM usage]]''' 
* When openAlex has finished indexing we will cut down the ARC to 8GB by changing a line in
 nano /etc/modprobe.d/zfs.conf
change arc maxline to
 options zfs zfs_arc_max=8589934592
save and close then run
 update-initramfs -u
 echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_max
 cat /sys/module/zfs/parameters/zfs_arc_max
It's a good idea to confirm that the kernel is actually respecting that 16GB cap you just baked into the initramfs. '''Note''' it should show the number of bytes not the number of KB or GB
 arc_summary | grep -A 5 "ARC Size"
 # alternatively, to get the ARC size directly from the kernel's statistics
 grep c_max /proc/spl/kstat/zfs/arcstats

==Secondary Failure: The Network Blackout==

On reboot Tayberry could not ping any host outside of Kiwi, but Kiwi could ping other hosts and they are sharing the same physical cable and vmbr0 so not a physical problem. The only difference between Kiwi and Tayberry is that Kiwi is untagged and Tayberry is tagged to VLAN 100. Tayberry did have an IP address and the NIC was up all of the settings were correct like the VLAN aware vmbr0 and Tayberry was assigned to the correct VLAN. However, VLAN-aware bridge (vmbr0) on Kiwi did not automatically restore the tagging for the Trunk port (nic0) and the command on kiwi:-
 bridge vlan show
but it did not show the VLAN100 as it should have done with the software defined network and Tayberry's NIC being assigned to VLAN100 (as no tap200i0 devices were present). To restore the VLAN100 we used the command 
 bridge vlan add dev nic0 vid 100
Which did than allow Tayberry to ping the other VMs within VLAN100 on Pear. VLAN100 should have been added as part of the VLAN IDs in the GUI definition of vmbr0 and there should have bee a line in the file /etc/network/interfaces. To find out why it had had not been added properly we check the file with
 nano /etc/network/interfaces
and we noticed that the line about bridge vids was missing so we added the line underneath the "vlan aware" line
 bridge-vids 2-4094
 ifreload -a 
Assigning a VLAN to a VM in the Proxmox GUI only tags the virtual interface; if the physical bridge port (nic0) isn't configured as a trunk (via bridge-vids), it will prune the tagged packets before they reach the physical wire. With these changes in place Tayberry was able to ping any IP addresses on VLAN100. The next problem was that Tayberry would not reach the dns server or any other VLAN. On Tayberry, using the command
 ip route
Showed that the default rout had been dropped. As Tayberry is a Debian server build it has old style networking so the interface is defined in /etc/network/interfaces the easiest way to restore the gateway is to use the command:
 sudo ip route add default via 192.168.100.1 dev eth0
With this finally done we could ping blackberry or any other host on pear. To make sure the default rout definition survives a reboot check the file
 /etc/network/interfaces
Under iface eth0 inet static, ensure there is a line: gateway 192.168.100.1

==The Third problem Service Recovery & OpenSearch Integrity ==

When the ZFS HDDs locked up there was possibly going to be some potential data corruption or "Stale File Handles" on the direct-mounted 5TB drive (/dev/sda) following the hard reset. However, because the direct-mounted drive (/dev/sda) was formatted with a journaling file system (or ZFS), and we performed a Sync (S) during the REISUB, the OpenSearch data remained consistent and the cluster returned to GREEN immediately. 

The Solution
* Health Check: Verified cluster status as GREEN using the OpenSearch Health API.
* Resume Indexing: Modified the indexing script to jump to file 1274 (to account for the ~341M records already committed) and used ionice -c 3 to prevent the indexer from hogging disk bandwidth again.
* '''Note:''' Resuming at file 1274 may result in an 'Overlap Zone' where the indexer processes records already in the database. OpenSearch handles this via versioning; the _count will not increase until the indexer reaches truly new data.


==📝 Final Documentation Table==



{| class="wikitable" 
|+ 
|-
! Failure Layer	!! Symptoms	!! Root Cause !! Permanent Fix
|-
|Hardware || BIOS "Bootable drive" error !! Cold boot confusion ||	Reset boot order in BIOS
|-
|Host OS || Blinking cursor / Lockup || ARC Exhaustion (60GB/64GB) || Capped ARC to 16GB in zfs.conf.
|-
|Network || Tagged traffic dropped || Missing bridge-vids on Trunk || Added bridge-vids 2-4094 to Kiwi interfaces.
|-
|Guest OS || No internet/DNS || Gateway route dropped || Added gateway to Tayberry interfaces file.
|-
|App || Stopped Indexing || Host Crash || Resume via script with ionice -c 3.
|}