Photography Pipeline: The Pipeline Begins

This post is mainly background on the project. I'm writing some of these posts retroactively, and without regard to the actual chronology.

A hobby of mine is photography. I've got a Sony a6000 and lenses worth more than the camera body (The new body I want, the a6400, has had its production put on hold due to reasons). I've been wanting something not Google Photos where I can put my photos for sharing, archiving, and general organization. My basic goals when setting out:

  • Be able to bring in photos from Google Takeout, my random file shares that have photos on them I've collected over the years, and from my phone when I'm not home
  • De-duplicate photos from the various sources
  • Attach geolocation to photos
  • Able to tag and search by photo metadata
  • Find similar photos
  • Have photos still be browsable without any application, and keep track of the various photo metadata added.
  • Be able to self host, or host on Azure

For tagging and organization I'm picturing something along the lines of danbooru.

Photo Ingress

Currently, my two big stores of photos are OneDrive and Google Photos. Getting them out of Photos in some automated way would probably be difficult, and I have other plans for photos taken after this system starts working. So, I went and requested a Takeout of my photos, resulting in four 50gb tgz archives. These were all put onto a dedicated zfs volume on my server so it will be easy to clean up once all this is done.

There's one complication with the data on Google Photos: For some reason Samsung devices shove screenshots into the DCIM directory where they get picked up by the Photos backup and there's no way to change this behavior on either the Photos or Samsung side. Dropping a .nomedia file into the screenshot folder would result in the screenshots being unavailable in the media picker. I've worked around this with a combination of a .nomedia file and a program to copy things placed into the screenshot folder elsewhere. But there's still years of screenshots in photos.

Now, for new photos, I have a plan. I'm already using an Android app called FolderSync to copy photos off a USB-C SD card reader to my phone because the wifi copy on my camera is annoying and slow. FolderSync also supports WebDAV. I found a nice Docker image of nginx configured to use WebDAV of dgraziotin/nginx-webdav-nonsense and set this up on my home server. I then just added its endpoint to the HAProxy instance on my firewall that handles SSL termination and redirecting to various hosts and ports around the house based on hostname. With that, getting the files to somewhere I can do something with them is solved.

social