UpLoad Server

Introduction

We need a method of easily getting image files uploaded from both LAN and Internet ready to be edited and displayed on the Webservers. For most of the webservers there will not be too many images to be uploaded so direct access with a FTPS server on the webserver host will be good enough. However, for the Plum (Photo) there will be a significant amount of images to be uploaded, it is after all the Piwgo photo webserver and some pre processing and backups will be needed. We expect to do the backups from Backup Server. There will be some changes to the backup server/plum webserver to reverse the storage we will have to change the documentation for them as well.

Photo Management

The general idea is to have an upload server as a staging point to get rid of duplicates, organise metatags and maybe enhance some of the photos and eventually transfer to the Piwigo website on Plum (Photo), we will separate these duties on to this server. To make it all a self contained object we will make this a desktop Unbuntu install rather than the usual server installs. It is uncertain if that is going to be fast enough to actually edit photos but it should be good enough to edit any metatags.

Workflow Outline

The first thing to do is to dump all photos on the physical desktop PC on the 4tb drive it doesn't matter if there are duplicates as they will be addressed as part of this workflow. The important thing is to make sure there are none missing. Once there are photos to upload simply copy them to Satsuma via the SMB shares Import Export, they have been mapped to a pair of network drives (S: is Import and T: is Export). These network shares are on SMB so will only ever work on the LAN and they have been blocked from the internet. The photos are on Satsuma dir /mnt/images/Import and /mnt/images/Export. Once the files are on Satsuma they will be checked for duplicates and previously known, all new unique files will be moved to a separate input directory for processing by Digikam. When there are a few photos the user can login to the plum desktop and start Digikam (I was going to use Shotwell but it would appear that Digikam is a better fit). Digikam should pull the photos an copy them in to its own directory. The user edits the photos or adds tags to them and when finished they should be exported to another staging area to be sent to Plum (Photo). While this leaves us with several copies of the same photos we should be able to have a script periodically delete the various copies, as a long term feature the photos will be left on the Digikam working directory (it is likely that Digikam will show loads of errors if it's working directory is cleared). The running of the various scripts will be tracked with some XML log files to cope with errors. The main processes flow will be tracked by a full database on Mandarin the MySQL Server. Once Photos are added to the final staging area a script will copy them on to Plum (Photo). As they appear on Plum the photos can be displayed or not as defined by the Piwigo software. Also the photos will be copied by the Backup Server Strawberry from a NFS share from Plum to Offsite backup.

Satsuma Setup

Satsuma is a Ubuntu desktop install to allow for some graphical applications to be run, we will have to see how well that runs.

Samba Install & A ZFS Hard Drive

To allow photos to be uploaded in the simplest method is to use SMB protocol and then setup network drives on any windows PC. It should be noted that SMB is not very secure so it should only be allowed on the LAN and make appropriate rules in the firewall. To install Samba log in to the host Satsuma

sudo apt update
sudo apt install samba

Setup a drive for samba to use by adding a separate drive in Proxmox from one of the ZFS datasets. Then start and login to the VM Satsuma. once we are logged in we need to identify the disk so run

lsblk

Look through the listings for the block devices to identify the new disk. Look for the new disk (e.g., /dev/sdb). The output will show the UUID (e.g., UUID="a1b2c3d4-e5f6-7890-1234-567890abcdef" Use fdisk to create a partition on the disk.

sudo fdisk /dev/sdb 

Replace /dev/sdb with your disk. Inside fdisk:

• n (New partition)
• p (Primary partition)
• 1 (Partition number 1)
• Enter (Default first sector)
• Enter (Default last sector)
• w (Write changes and exit)

When that is done we need to format. To Format the partition with ext4 filesystem:

sudo mkfs.ext4 /dev/sdb1 # Replace /dev/sdb1 with your partition

We will now need to crate the mount point

sudo mkdir /mnt/images

To use the partition UUID to mount it:

sudo mount UUID="YOUR_PARTITION_UUID" /mnt/images 

1. Replace with your YOUR_PARTITION_UUID

The mount like this will disappear on reboot so to make the mount persistent we would need to edit the /etc/fstab

sudo nano /etc/fstab

WE need to add the line UUID="YOUR_PARTITION_UUID" /mnt/images ext4 defaults 0 2 # Replace with your UUID Save and close the file now the drive is set and formated we need to add the dirs for Samba to use

sudo mkdir /mnt/images/import
sudo mkdir /mnt/images/export

if the group photoshare has not been created yet

sudo groupadd -g 1002 photoshare

If the users aren't yet created

sudo useradd -u 1001 -g photoshare -m photoup

• -u 1001: Sets the UID to 1001
• -g photoshare: Sets the primary group to photoshare
• -m photoup: Creates the user's home directory.

sudo passwd photoup (set a password for the user)

add the users to photoshare

sudo usermod -aG photoshare nigel
sudo usermod -aG photoshare photoup

If you need photoup to have sudo privileges for installing software or system administration:

sudo usermod -aG sudo photoup

Then we can set the permissions of the drive

sudo chown nigel:photoshare /mnt/images/export
sudo chown nigel:photoshare /mnt/images/import

and allow full access

sudo chmod 765 /mnt/images/import
sudo chmod 765 /mnt/images/export

now that the users are ready and the had drive has been added and configured we can setup Samba

sudo nano /etc/samba/smb.conf

Scroll down towards the bottom and add [Import]

       comment = Share for importing files
       path = /mnt/images/import
       browseable = yes
       read only = no
       guest ok = no
       valid users = nigel, photoup # Users with import permissions
       create mask = 0664 # Permissions for new files
       directory mask = 0775 # Permissions for new directories

[Export]

       comment = Share for exporting files
       path = /mnt/images/export
       browseable = yes
       read only = yes # Export share is read-only
       guest ok = no
       valid users = nigel, photoup # Users with export permissions

Save and close the file then restart Samba

sudo systemctl restart smbd

Set the password within Samba

sudo smbpasswd -a photoup
sudo smbpasswd -a nigel

We will now need to add a rule to Pfsense in fact two rules, one for port 139 and another for 445. We should create aliases for the relevant values. so logon to Pfsense and go to firewall->aliases and add the following

• alias IP_Samba should be set to the IP Address of the Samba server this would be the host satsuma
• alias Samba_high is set to the port 445
• alias Samba_low is set to 139

We will need to now add the two rules for Samba. Logon to the Pfsense and select Firewall->Nat and click the add a rule button. The rule should be

• Interface should be WAN
• Protocol is TCP
• Source should be network and 192.168.0.0/24
• Source Port should be any
• Destination should be This firewall (self)
• Destination Port Range is Other->Samba_Low to Other->Samba_low
• Redirect Target IP should be Address or alias->IP_Samba
• Redirect Target port should be Samba_Low
• Description should be something like Forward samba to samba server
• NAT reflection should be Enable (NAT + Proxy
• Filter Rule association should be create new associated filter rule ( note this will change when the rule is written to be the same as the comment)

Then save. Create another rule exactly the same but with Samba_high as the port.

We are now ready for testing in Windows 11 Pro. On the desktop of the PC open the start menu and type notepad then open it with administrator.

• Select open file
• In the open dialog select the address C:\Windows\System32\drivers\etc
• In the file type box select all files
• select hosts
• scroll down to the bottom of the file and add the line

192.168.0.x   photo # add the actual IP address of the Pfsense WAN port and if a different name to photo if required

• save as the file then change the file type to All files (*.*)
• from the Start menu type cmd and open as administrator
• type net use T: \\Photo\Export /user:satsuma\photoup photoupsPassword /persistant:yes ( this will map a network drive to T: the share will be at photo and the directory will be Export (it will be case sensitive and must match the [Export] block in Samba.conf), Photo will be whatever you put in hosts or could be changed to the IP address of the WAN port of Pfsense the user will be the Samba user and that must be on satsuma the password will also come from satsuma, if the hostname is less desirable then satsuma's ip address could be used, Persistent:Yes means the drive will be persevered on reboot of the PC.
• type net use S: \\Photo\Import /user:satsuma\photoup photoupsPassword /persistant:yes ( this will map a network drive to T: the share will be at photo and the directory will be Export (it will be case sensitive and must match the [Export] block in Samba.conf), Photo will be whatever you put in hosts or could be changed to the IP address of the WAN port of Pfsense the user will be the Samba user and that must be on satsuma the password will also come from satsuma, if the hostname is less desirable then satsuma's ip address could be used, Persistent:Yes means the drive will be persevered on reboot of the PC.
• Open Windows Explorer
• Browse to This PC and you should see two new drives one called Import and the other called Export. You should be able to add files to Import but Export will be just for export so will not be readable.

Setting up the Scripts

It is likely that there will be duplicate files uploaded because there are duplicate file stores that have been used over the years. To eliminate these duplicate files we will do a SHA256 on each after it has been uploaded to the import directory and compare the result to a database of previously stored files. while this sound really simple there will be some complexities that will be discovered along the way. The first obvious hurdle is what to do with any duplicates found I think the best solution will be to move them to different dirs depending on what their status is.

Directories and their uses

To keep things organised we will have several directories on the Images partition. All of the following will be dirs that have /mnt/images/ as their root

• import will be where Images are uploaded to as they come in to the system it will be messy and chaotic with the potential of having loads of duplicate files in different dirs.
• export will be for export of probably other files not related to the photo store.
• duplicates be a dumping ground for duplicates found prior to them being discarded.
• working will be a where each file will be moved to while it is being processed this dir will only ever have one file in it.
• digkam_in this will be where Digikam imports images from. By the time it files get here they will be known to be non duplicates. There will potentially be a lot of files here all with unique name but all in the same dir so a listing here could be of thousands of files

Processing with Digikam

I will process the