AI & Jellyfin

Introduction

The objective for Quince is to make it the power house Virtual Machine for the AI so it needs to have the Nvidia 5060TI GPU passthrough completed . At the same time it will also run a media server in the form of Jellyfin so that the GPU can do the transcoding. With the AI being serviced by this host we can use Blackberry for the Data Harvesting.

Docker Applications installed on Quince

We are going to need to install several applications that will share the GPU. The first will Dockge so that any new containers can be managed easily. We will also need to install Ollama so that we can run LLMs easily. Then to make use of the data archive we can use AnythingLLM.

Installation Strategy

Once the Blackwell GPU passthrough was verified on the Pear host, we transitioned to the Quince VM to set up the containerized environment. This allows us to run high-performance AI (Ollama) and media (Jellyfin) apps while keeping the base OS clean.

Docker Engine Installation

We use the official Docker repository to ensure access to v29+, which includes critical patches for Gen 5 PCIe and Blackwell architecture support.

sudo apt update
sudo apt install ca-certificates curl gnupg

Then setup the repository

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Next Install Engine & Compose

sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

NVIDIA Container Toolkit (The "Magic Bridge")

This toolkit enables the libnvidia-container library, which maps the physical GPU device files (/dev/nvidia0, etc.) into the virtualized Docker namespace.

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit

and last we configure the Nvidia Container tool Kit and restart Docker

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Docker Applications

Dockge Setup

Now that docker is installed we can install Dockge. To get Dockge running on Quince, we followed a specific path that keeps it isolated but powerful enough to manage your other stacks (Jellyfin, AnythingLLM, etc.). Since Dockge is a "Docker manager that runs inside Docker," the installation is a bit like a "Inception" move, we create a directory for it, and then use a simple script or a compose.yaml to launch it.

• Create the Directory Structure On Quince, we created a dedicated space for Dockge and all the future stacks you'll build. open a terminal and run the following

mkdir -p /mnt/docker_data/dockge /mnt/docker_data/stacks
cd /mnt/docker_data/dockge

• Download the Compose File We pulled the official configuration. Dockge needs access to the Docker Socket (/var/run/docker.sock) so it can "reach out" and control the other containers on Quince.

curl https://raw.githubusercontent.com/louislam/dockge/master/compose.yaml --output compose.yaml

• Launch Dockge We started it up in "detached" mode.

docker compose up -d

Now that Dockge is running we have a web interface on Quince:5001 and we can use it to add new services.

Jellyfin Installation

To give the media server plenty of storage we have mounted a 3TB data drive at /mnt/jellyfin/ to store all of the media files, we have another drive at /mnt/docker_data to hold all of the configuration data for the containers. We had some difficulty with a left over jellyfin container instalation so we called this container jellyfin-new. The yaml file for "jellyfin-new" is as follows.

services:
  jellyfin:
    image: jellyfin/jellyfin
    container_name: jellyfin
    user: 1000:1000
    # ADD THIS SECTION:
    runtime: nvidia # Tells Docker to use the NVIDIA Container Toolkit
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu, video] # Grants access to NVENC/NVDEC
    ports:
      - 8096:8096
    volumes:
      - /mnt/docker_data/jellyfin/config:/config
      - /mnt/docker_data/jellyfin/cache:/cache
      - /mnt/jellyfin/audiobooks:/data/audiobooks
      - /mnt/jellyfin/oldfilms:/data/movies
      - /mnt/jellyfin/oldseries:/data/tvshows
      - /mnt/jellyfin/dji:/data/dji
    restart: unless-stopped
networks: {}

The web interface for Jellyfin is on port 8096 so http://quince:8096 will bring it up. To setup the local devices either with a webrowser or the Jellyfin Media player the firewall will forward 8096 to the media server.

AI applications

We want to be able to run a variety of LLMs so will setup Ollama with a container named ollama and a yaml file as follows

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    volumes:
      - /mnt/docker_data/ollama:/root/.ollama
    networks:
      - ai-network
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities:
                - gpu
    restart: unless-stopped
networks:
  ai-network:
    external: true