The Kiwix Archive - Revision history

Wikisailor: /* 📖 Introduction */

2026-04-20T20:29:48Z

📖 Introduction

← Older revision		Revision as of 20:29, 20 April 2026
Line 4:		Line 4:
	* The Goal: To provide a permanent, offline knowledge base that remains accessible even if the internet is down, serving everyone on your local network.		* The Goal: To provide a permanent, offline knowledge base that remains accessible even if the internet is down, serving everyone on your local network.
	* Synergy: Works alongside '''[[OpenAlex]]''' (scholarly search) and '''[[The Web Archive (ArchiveBox) \| ArchiveBox]]''' (personal web snapshots) to create a three-tier local research library.		* Synergy: Works alongside '''[[OpenAlex]]''' (scholarly search) and '''[[The Web Archive (ArchiveBox) \| ArchiveBox]]''' (personal web snapshots) to create a three-tier local research library.

			Once this collection of Zims are downloaded we can also extract the text from the files to create a collection of .MD files which can then be used as a Retrieval-Augmented Generation database or RAG database for offline LLMs or the same data could be used as fine tuning data for o0ne or more LLMs. Although some of the .MDs will be general knowledge and only really useful for the RAG DB some will detailed question and answer tables of condensed information for example the Slack Overflow will be high density data sets

	==💾 The Infrastructure ==		==💾 The Infrastructure ==

Wikisailor: /* 💾 The Infrastructure */

2026-03-03T09:45:34Z

💾 The Infrastructure

← Older revision		Revision as of 09:45, 3 March 2026
Line 10:		Line 10:
	* VM Config: Debian \| 4 Cores \| 6GB RAM.		* VM Config: Debian \| 4 Cores \| 6GB RAM.
	* Storage: 4TB XFS disk mounted at /mnt/docker_data and an additional 5TB XFS disk for ArchiveBox /mnt/archive_data		* Storage: 4TB XFS disk mounted at /mnt/docker_data and an additional 5TB XFS disk for ArchiveBox /mnt/archive_data

			==[[Linux Commands]]==

			A set of Linux commands to help show the progress of indexing and file copying

	==🐋 The Software Stack (Docker)==		==🐋 The Software Stack (Docker)==

Wikisailor: /* 📖 Introduction */

2026-02-09T09:00:12Z

📖 Introduction

← Older revision		Revision as of 09:00, 9 February 2026
Line 3:		Line 3:
	* The Format: It uses highly compressed .ZIM files. A single file can contain the entirety of Wikipedia (with images) or the complete medical encyclopedia.		* The Format: It uses highly compressed .ZIM files. A single file can contain the entirety of Wikipedia (with images) or the complete medical encyclopedia.
	* The Goal: To provide a permanent, offline knowledge base that remains accessible even if the internet is down, serving everyone on your local network.		* The Goal: To provide a permanent, offline knowledge base that remains accessible even if the internet is down, serving everyone on your local network.
	* Synergy: Works alongside OpenAlex (scholarly search) and ArchiveBox (personal web snapshots) to create a three-tier local research library.		* Synergy: Works alongside '''[[OpenAlex]]''' (scholarly search) and '''[[The Web Archive (ArchiveBox) \| ArchiveBox]]''' (personal web snapshots) to create a three-tier local research library.

	==💾 The Infrastructure ==		==💾 The Infrastructure ==

Wikisailor: /* 🌐 Accessing and Using the Library */

2026-02-09T08:06:55Z

🌐 Accessing and Using the Library

← Older revision		Revision as of 08:06, 9 February 2026
Line 77:		Line 77:
	\|-		\|-
	\| Kewix org \|\| https://download.kiwix.org/zim/ \|\| More Lists of Zim files		\| Kewix org \|\| https://download.kiwix.org/zim/ \|\| More Lists of Zim files
			\|}

	===Indexing Files after download===		===Indexing Files after download===

Wikisailor: /* 🌐 Accessing and Using the Library */

2026-02-09T08:06:26Z

🌐 Accessing and Using the Library

← Older revision		Revision as of 08:06, 9 February 2026
Line 77:		Line 77:
	\|-		\|-
	\| Kewix org \|\| https://download.kiwix.org/zim/ \|\| More Lists of Zim files		\| Kewix org \|\| https://download.kiwix.org/zim/ \|\| More Lists of Zim files

			===Indexing Files after download===

			There are three helper scripts in the /mnt/docker_data/stacks/kiwix-archive/zim/ directory that will help add new zim files to the xml index
			{\| class="wikitable"

			\|-
			! Filename !! Purpose
			\|-
			\|./audit_incomplete.sh \|\| scans zim dir for incomplete files and move them to incomplete directory
			\|-
			\|./sweep_parts.sh \|\| Checks any files in the incomplete directory for files that have corresponding completed downloaded Zim
			\|-
			\|./index_vault.sh \|\| Adds new Zim to XML index file
			\|}

Wikisailor: /* Installing Docker & Compose */

2026-02-09T07:48:29Z

Installing Docker & Compose

← Older revision		Revision as of 07:48, 9 February 2026
Line 18:		Line 18:
	# Update and install dependencies		# Update and install dependencies
	sudo apt update && sudo apt install -y ca-certificates curl gnupg		sudo apt update && sudo apt install -y ca-certificates curl gnupg

	# Add Docker’s official GPG key		# Add Docker’s official GPG key
	sudo install -m 0755 -d /etc/apt/keyrings		sudo install -m 0755 -d /etc/apt/keyrings

Wikisailor: /* Installing Docker & Compose */

2026-02-09T07:48:16Z

Installing Docker & Compose

← Older revision		Revision as of 07:48, 9 February 2026
Line 23:		Line 23:
	sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc		sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
	sudo chmod a+r /etc/apt/keyrings/docker.asc		sudo chmod a+r /etc/apt/keyrings/docker.asc

	# Add the repository to Apt sources		# Add the repository to Apt sources
	echo \		echo \
Line 29:		Line 28:
	$(. /etc/os-release && echo "$VERSION_CODENAME") stable" \| \		$(. /etc/os-release && echo "$VERSION_CODENAME") stable" \| \
	sudo tee /etc/apt/sources.list.d/docker.list > /dev/null		sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

	# Install Docker Engine and Compose Plugin		# Install Docker Engine and Compose Plugin
	sudo apt update		sudo apt update
	sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin		sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

	# Optional: Allow your user to run docker without sudo		# Optional: Allow your user to run docker without sudo
	sudo usermod -aG docker $USER		sudo usermod -aG docker $USER

Wikisailor: /* 🐋 The Software Stack (Docker) */

2026-02-09T07:46:52Z

🐋 The Software Stack (Docker)

← Older revision		Revision as of 07:46, 9 February 2026
Line 12:		Line 12:

	==🐋 The Software Stack (Docker)==		==🐋 The Software Stack (Docker)==

			=== Installing Docker & Compose===

			Before installing Dockge, we must install the Docker engine and the Compose plugin officially on Debian.
			# Update and install dependencies
			sudo apt update && sudo apt install -y ca-certificates curl gnupg

			# Add Docker’s official GPG key
			sudo install -m 0755 -d /etc/apt/keyrings
			sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
			sudo chmod a+r /etc/apt/keyrings/docker.asc

			# Add the repository to Apt sources
			echo \
			"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
			$(. /etc/os-release && echo "$VERSION_CODENAME") stable" \| \
			sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

			# Install Docker Engine and Compose Plugin
			sudo apt update
			sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

			# Optional: Allow your user to run docker without sudo
			sudo usermod -aG docker $USER

	===🛠️ Installing Dockge===		===🛠️ Installing Dockge===

Wikisailor: /* 🌐 Accessing and Using the Library */

2026-02-09T07:39:18Z

🌐 Accessing and Using the Library

← Older revision		Revision as of 07:39, 9 February 2026
Line 52:		Line 52:
	\|Kiwix Web UI \|\| http://blackberry:8080 \|\| Browsing your downloaded offline libraries.		\|Kiwix Web UI \|\| http://blackberry:8080 \|\| Browsing your downloaded offline libraries.
	\|-		\|-
	\|ZIM Library \|\| kiwix.org/en/download \|\| Where to download new content (Wikipedia, StackOverflow, etc.).		\|ZIM Library \|\| https://kiwix.org/en/download \|\| Where to download new content (Wikipedia, StackOverflow, etc.).
	\|-		\|-
	\| German Zim Library \|\| https://ftp.fau.de/kiwix/zim/ \|\| lists of zim files.		\| German Zim Library \|\| https://ftp.fau.de/kiwix/zim/ \|\| lists of zim files.
	\|-		\|-
	\| Kewix org \|\| https://download.kiwix.org/zim/ \|\| More Lists of Zim files		\| Kewix org \|\| https://download.kiwix.org/zim/ \|\| More Lists of Zim files

Wikisailor: /* 📄 Kiwix YAML (The Stack) */

2026-02-09T07:38:51Z

📄 Kiwix YAML (The Stack)

← Older revision		Revision as of 07:38, 9 February 2026
Line 42:		Line 42:
	restart: unless-stopped		restart: unless-stopped
	networks: {}		networks: {}

			===🌐 Accessing and Using the Library===

			{\| class="wikitable"

			\|-
			! Tool !! URL !! Purpose
			\|-
			\|Kiwix Web UI \|\| http://blackberry:8080 \|\| Browsing your downloaded offline libraries.
			\|-
			\|ZIM Library \|\| kiwix.org/en/download \|\| Where to download new content (Wikipedia, StackOverflow, etc.).
			\|-
			\| German Zim Library \|\| https://ftp.fau.de/kiwix/zim/ \|\| lists of zim files.
			\|-
			\| Kewix org \|\| https://download.kiwix.org/zim/ \|\| More Lists of Zim files