Refining my "Air Gapped" Backup/Archive, rather than a New Year’s Resolution

I’ve not been so diligent; my "air gapped" archive/backup system has been sitting on the shelf for the past two years, gathering dust. You know how it goes. I had good intentions about checking those old hard drives every six months (they need to be "exercised" to stay healthy), but eye issues and life got in the way.

When I finally fired it up again, I accidentally left the Wi-Fi on, and Ubuntu immediately reminded me about updates. At first, I'd been avoiding updates thinking "I barely use any features, why bother?" But now seemed like the perfect time to upgrade and add some new tasks into the system.

Enhancing the 3-2-1 Backup Strategy

This whole approach really came out of the ransomware era (still very much a security problem). The idea is simple: keep an offline backup so you can recover if something goes wrong, and make sure you're actually verifying that your stored data is correct zero errors.

 Just looking at a photo or document isn't enough to know if it's been corrupted or tampered with. How do you really check a file? It depends on the type of file and format, and there’s a growing miriad of different ones to worry about.

My Practical Approach

For photos, I'm using MD5 checksums, basically digital fingerprints for each file. Adobe has its own proprietary checksum system in Lightroom and new content credential tools, but I'd rather stick with the commonly used standard.

Raw files are trickier. They vary between camera manufacturers and have already changed over time and camera models.. Some photographers convert everything to lossless TIFF files (which can become huge), others swear by Adobe's DNG format, but I'm not seeing widespread adoption outside the Adobe ecosystem.

Video files are a more difficult area, they rapidly gobble up space and I've got formats from software I have no idea about. For now I will just have to stay with VLC video viewer, which seems to cover most video formats.

The Reality Check

I'm being realistic here, I’m getting older, and I have half a million photos. Nobody wants to wade through all of that when I'm gone. So instead of keeping everything organised just by date and time, I need to actually curate the mess and my legacy. Pull out the family photos, separate events and places, get rid of the junk. I'm planning to explore best ways to identify the high-value stuff worth preserving properly.

The plan is to do this reorganisation on my main computers first, then pass over a well-organised set of archives to the air-gapped system. It's a bit scary because I'll be deleting files, so I'm keeping the full backup for at least a year as a safety net. Currently sitting at 5.5 terabytes and hoping to cut my archive at least in half, but we'll see how realistic that is, same time next year.

MD5 Reminder

In our digital age, it has become increasingly important to protect our personal data and information from accidental loss, malicious individuals and cyber threats. Photos can hold a lot of sentimental value and unsuspected personal information. A simple way to ensure the integrity of our photos is by creating an MD5 hash code.

An MD5 hash code is a one-way encryption that produces a fixed-length string of characters that is unique to the photo, and supposedly a unique identifier. By checking an MD5 hash code for your photos, you can verify that the photo has not been tampered with or altered in any way.

My heart sank recently when I realised I had a significant difference in the archived directories and files saved during 2010. I have been using a couple of repurposed old harddisk, standing in for one set of my photo archives, and alas they are showing signs of aging, and probably slow failure. Maybe I’ve been backing up a corrupted or damaged digital photo set. I hadn’t even gone looking for photo there for a while.

Yet it wasn’t time to panic, I had actually gone back and generated an MD5 hash code list for each directory of files (one list per monthly directory) back in 2017. Despite having three versions of the archives (one possibly suspect) and significant differences in files, these MD5 directory lists were largely the same and when checked identified the missing files. Fortunately, there was always one version of these missing files, sometimes in a different directory,  and on checking they were photos I didn’t want to keep (out of focus, almost duplicate shots, overexposed etc.). There were just three photos in the end that may have gone astray. Trying to do this by visually browsing the pictorial version of the photos amongst such a large group of files would have been very difficult, if not almost impossible. So phew…

I didn’t find any corrupted files looking through the photos or the MD5 Checks but I am confident the hash code will identify such damaged files, but avoid overwriting older undamaged versions and unknowingly backing them up. I can breathe easy now...

PS Another advantage of creating an MD5 hash codes for photos is that it provides a way to detect if the photo has been tampered with or modified. For example a hacker might encrypt the photo in order to send you a ransome, the MD5 hash code will change, indicating that the photo has been altered. To beat the hacker you should already have a viable backup, but you will be able to avoid unknowingly backup his altered file(s) and could ignore his ransome request. 

Making A Hash Of Checksums

I come from a very deep computer background, (I first used computers in 1969) and have managed several mini computers (DG, PDP & VAX) and networks of PC so I am well grounded in the need for both Backup (keep a safe copy) and Archive (long term storage). Now if you read a lot of net advice you might think they are the same and just call that process backup. Backup is important, and a lot of people still don’t do it adequately and I have written a fair bit about it, but archiving seems to have escape my posting attention (other than my The Importance of Cataloging your Digital Photo Archive Post).  I need to rectify that and I will begin with a bit more detail on how I seek to avoid storing, and identifying corrupt file

MD5 checksums (aka hash values)

With the recent death of two western Digital 2GB external drives, I have become a bit paranoid about the potential for file corruption as a drive slowly fails (One drive just dropped dead the other slowly started to show problems). Because most of the files are in a binary format such corruption can easily go unnoticed and the corrupt file could easily be diligently backed up. There are solutions, like having mirrored drives (extra hardware & software and expense) or regularly reviewing the files (a challenges with lot of photos). I have found a much simpler approach, that is to use checksums created from each file. If the file is corrupted the check sum will change. So each time a file is moved or the disk is rotated these check sums can be checked. I choose MD5 because they are a public format and there are a lot of utilities to create them and they are widely used to detect duplicate files (particularly photos, video & music Files. It would be nice if photo software undertook this task automatically but alas while I can see similar numbers in picasa and lightroom they are not true MD% hash values but just look like them. My only conclusion is they are propriety formats, which creates the risk that if the company disappears (google has already washed its hands of picasa) so does the long term suitability as check sums.  The process to create these files can take some time and I am using Total commander, which creates the check sum for an entire subdirectory (folder) and writes a single .md5 file containing the check sums. This is an ASCII file and can easily be read or a particular Hash value cut and pasted into a different utility to verify check sums. They are small and take up negligible space. To verify the checksums also takes a little time because the checking software must read through each file. Good utilities will be able to read the .MDR file and report missing files or errors (ie corrupted files). Total commander has a second utility to do this job. So the final big question is – “how do I know the photos are ok, not corrupted, when I make the MD% hash value?” Simple answer is I can’t be sure so I also have to at least look at the files, using picasa, the default windows photo viewers, Lightroom or Corel Aftershot Pro (which actually seems the fastest option particularly with RAW files). So setting up the checksums for a proper archive does take some time but but I can have more confidence in that archive as it is passed around different media and locations.

The Importance of a Catalogue to your Digital Photo Archive

A shoe box or album full of photos is easy to flick through and you can quick see both its condition and content. Digital files however have limited visibility, especially a collection of digital photos. The Photo file will be in binary and unless you have specific software to decode it it will just be symbols and numbers at best. One exception is the jpeg format which is so widely used that most software, browsers, computer and smart phone apps with recognise and decently render it. Other formats and in particular the camera RAW formats will always require social software (and given that these formats are proprietary and there is a risk of them becoming obsolete) may not be guaranteed to be available on different hardware and/or in the future. Further It is unlikely that scanning the file names from the camera (like IMG_0007.CR2) will be very meaningful. You need a reader that understand the format of that data and can render it as an image. A decent catalogue (or Index) of you photos need both to be able to easily view you photos but it also benefit from some organization that makes the images quicker to find

So how do you figure out what is there. There are two common solution,

1. The first is Thumbnails, normally postage stamp size low resolution version that give you a peek at what is in the photo.
2. The second is to use metadata (data about the context of the photo itself).

There are lots of decent (and free or inexpensive thumbnail programs) like Picasa XnView, Photo Mechanics, the default folders view in windows even lets you select a small, medium or Larger icon view, which are thumbnails. Some can handle RAW formats but other may not (or rely on updating a codex for that format). Other software like Aftershoot and OnOne 10 have a browse mode so you don’t have to “import” photos first, you just get presented with a grid of thumbnails. These are fast and a joy to use compared with opening up directories of images in say Lightroom. Having a good thumbnail facility that is fast and easy to use is the first additional need for a good archive.

Metadata, (data about the photo and it’s contents) is a little more complex but it should not be. There are two standards EXIF and IPTC metadata formats. The EXIF data is really mainly about the camera and image and has been well addressed by camera and phone manufacturers (although they may not record all the possible metadata). This EXIF metadata is included with the standard jpeg data format so the metadata travels with the image embedded in it. The IPTC Interchange Model is a way to record information about who created the photo and how it is licenced for distribution and publication (it was set up for exchange of information, and images between newspapers  and is not a big deal in a personal photo archive). This information can be, but seledom is, embedded in jpeg files. Most software that works with photo will let you display both forms of this metadata. If you are having trouble reading the metadata Phil Harvey has written a wonderful open source program called ExifTool, which can interact with metadata in many formats (eg, embedded in the jpeg or stored in an .xmp sidecar file)

Whilst you can see a scratch of water stain on an old photo. It is tricker to judge damage too (corruption of a digital file) because in its raw binary digital format it if not easily comprehended. Even worse even a little damage may render the digital image unusable. There are technique to ensure digital files are not corrupted, most notable the checksums and Hash function (they are not hash tags). Of these MD5 is the most used for photos and is usually the technique of choice applied for detecting duplicates. Whilst I can see some software uses checksums (see picasa.ini of right) they do not appear to be MD5 numbers. Adobe use two different IDs which look a little like MD5 hash values, the first is a unique image ID created when The image is scanned by lightroom (or other adobe utilities) and the seconds is an instance ID and is updated whenever the files is accesses and parametrically adjusted (ie the .XMP file and/or catalogue are altered). I can not fnd any definitive documentation on them So I’m just going to assume they are yet another propriety secret and thus I doubt I will research them any further.

Fortunately there are lots of well tried and tested MD5 utility and programs, many free to download. What I discovered is the best way to create these hash table is not one for each file but with one file for each sub-directory. I already had a program called total commander (and alternative to windows explorer) which does this. Further you just need to click on that file and then Total commander will scan the files mentioned and report back if the hash value matches. Yes I did a test truncating a photo and the checked reported a hash value difference. There are a number of utilities so I assume you will have no problems finding a similar one.

I have been using this approach for the past 6 months it very simple and I just create the Hash Table for each directory when I do my monthly backup checks. I have been reprocessing some older directories as I move to a hard disk based archive. Running the hash function did Identified 2 photos that had read problems, luckily I had another copy of those which was fine. I am now slowly adding hash table to all my archived photo folders.

I’m now even more certain you need to also keep a checksum or hash value, as well as an easily accessed thumbnails and standard metadata utilities in addition to your archived files. Copies of the applications used to created these views and information are an important part of your catalogue and should be stored with your archived photos.