Refining my "Air Gapped" Backup/Archive, rather than a New Year’s Resolution

I’ve not been so diligent; my "air gapped" archive/backup system has been sitting on the shelf for the past two years, gathering dust. You know how it goes. I had good intentions about checking those old hard drives every six months (they need to be "exercised" to stay healthy), but eye issues and life got in the way.

When I finally fired it up again, I accidentally left the Wi-Fi on, and Ubuntu immediately reminded me about updates. At first, I'd been avoiding updates thinking "I barely use any features, why bother?" But now seemed like the perfect time to upgrade and add some new tasks into the system.

Enhancing the 3-2-1 Backup Strategy

This whole approach really came out of the ransomware era (still very much a security problem). The idea is simple: keep an offline backup so you can recover if something goes wrong, and make sure you're actually verifying that your stored data is correct zero errors.

 Just looking at a photo or document isn't enough to know if it's been corrupted or tampered with. How do you really check a file? It depends on the type of file and format, and there’s a growing miriad of different ones to worry about.

My Practical Approach

For photos, I'm using MD5 checksums, basically digital fingerprints for each file. Adobe has its own proprietary checksum system in Lightroom and new content credential tools, but I'd rather stick with the commonly used standard.

Raw files are trickier. They vary between camera manufacturers and have already changed over time and camera models.. Some photographers convert everything to lossless TIFF files (which can become huge), others swear by Adobe's DNG format, but I'm not seeing widespread adoption outside the Adobe ecosystem.

Video files are a more difficult area, they rapidly gobble up space and I've got formats from software I have no idea about. For now I will just have to stay with VLC video viewer, which seems to cover most video formats.

The Reality Check

I'm being realistic here, I’m getting older, and I have half a million photos. Nobody wants to wade through all of that when I'm gone. So instead of keeping everything organised just by date and time, I need to actually curate the mess and my legacy. Pull out the family photos, separate events and places, get rid of the junk. I'm planning to explore best ways to identify the high-value stuff worth preserving properly.

The plan is to do this reorganisation on my main computers first, then pass over a well-organised set of archives to the air-gapped system. It's a bit scary because I'll be deleting files, so I'm keeping the full backup for at least a year as a safety net. Currently sitting at 5.5 terabytes and hoping to cut my archive at least in half, but we'll see how realistic that is, same time next year.

DNG and the Month of Living Dangerously

Well i’m glad September is over, I can reset the dial of my beloved Pentax back to RAW/PEF not RAW/DNG. This story began several months ago when I installed the new version of OnOne 10.5, and suddenly I started having problems with OnOne Browse not display RAW files. Initially it wasn’t every RAW file but as I tried reinstalling the update and specifically once I had installed version 10.5.1 I was not able to view any .pef (the Pentax native RAW file) other than for a fleeting moment when the file was first viewed. Also .tif files and .cr2 first would intermittently disappear. This made trying to use OnOne software standalone tedious and frustrating. My continued request for support were just answered with trite rubbish about updating drivers and re-installing the software. I got frustrated and stopped using OnOne for a long time.

I got used to using Aftershot Pro mainly for my initial review and perhaps quick processing in the field. Then loading up lightroom back home and finishing images. For a few photos I wanted to do extra post processing I used the plug-ins linked to either OnOne Enhance, OnOne Effects or Nik’s Software.There is really nothing wrong with Aftershot Pro, in fact despite OnOne’s claims to have the fastest RAW rendering I suspect aftershot might be faster. However I quiet liked being able to flag just the images I wanted to further post process and load only those into lightroom or just finish them in the other OnOne modules, as I had been able to in version 10.0.1.  This was ok and seldom more than two steps. usinf aftershot, then lightroom and then a plg-in was three and meant having to generate an intermediate exchange file. Then I remember that my Pentax is one of the very few cameras that can capture DNG files in camera. I had previously looked at using that format but the files where substantially bigger so I couldn’t see any benefit. However if I avoided the intermediate copies I felt I could still be ahead in terms of spaced consumed so I bravely decided to only take DNG files during September.

Where did all the space go?

 

In september I created 1420 photo files. Most RAW plus jpeg pairs and most of them RAW DNG with a few .cr2 files. They took up a whopping 85 GB (excluding any .tiff post processing versions)
compared with 3710 files taking up only 40gb for my  month long candian month long trip earlier.

Exploding DNG  Grenades

The first hint of an issue had showed up before September but I assumed it was an isolated accident. Occasionally, and I still can not figure out why or which piece of software is to blame, some of the DNG files will suddenly double in size. Typically it is my better shot that I am interested in. Then sometimes they double in size again. as an example most DNG file from the camera are around 23 MB each, then when they first explode they will reach around 54MB then the second explosion will see them at 120mb plus. It is only when I try to view one of these exploded files that the havoc begins. Aftershot nicely reports it has stopped working, and trying to restart it report that it shut down unexpectedly and is set to read that directory again but it nicely reports it is stopping again, and so on. Lightroom seems to disappear for a while and eventually just doesn’t acknowledge that these bigger DNG files exist. OnOne browser however goes into suspended animation and my computer totally locks up. If I kill the process, restart the computer OnOne will refuse toit refuses to restart.

I eventually worked out if I renamed the last directory I had used before such crashes I could restart the misbehaving software. However if I tried to re-access that directory with OnOne browse, it was back to sleep, and crashes again. Lightroom just kept ignoring the big files after long contemplation and aftershot was ok unless I tried to look more closely at those files. Picasa would still view these files but took a long time to load them, perhaps understandably because they were large. These problems occurred on both my new HP Spectre laptop and my HP desktop, but less so on tests on an old Toshiba laptop.

I soon came to the conclusion that these exploded DNG files were not good, possibly corrupted some how. Which left me with a risk, if I tried to view these files before I had a backup I could loose the RAW version (since I take RAW and jpeg I would at least still have the jpeg version). Unfortunately I realised this AFTER I lost a couple of rather nice photos. I do nothave decent DNG version of these now.

As the month progressed this just became a bigger and more frequent problem and I am no closer to knowing why. I just have a mass of DGN files I am reluctant to even veiw.

Toxic Tiffs

When I am working with plug-ins like OnOne or Nik’s software I had adopted the idea of using .tiff files which give me more colour bit depth and preserve the illumination dynamic range compared with Jpeg and are way smaller than using the .png format (photoshop file format). Again occasionally to start with these .tiff files derived from DNG files began to refuse to load. I could work on the file to begin with, but once i quit the plug-in concerned the file would not open. Worst once this occurred within a session all other tiff files I tried to access would suffer the same fate. Exiting the software and starting a new session sometimes fixed the problem till I hit another toxic .tiff. the file sized does not seem to be altered in the case of the .tif files, so there is no easyway to figure out which fie will or won’t open. So opening any .tif file has a risk!

At this stage I also started to suspect a photo/dng targeting virus or malware, but I can not findany reports of such a beast and my virus and malware scans do not report any problems,

So I have stopped recording in DNG for four days now and guess what? no problems for two days!

 

Is the problem in the Pentax version of DNG, or is it in the software, or even combination of software I’m trying to use?

 

The fact that the new apple IPhone 7 will save optionally save in DNG format could see others finding similar problems. I’ll let the do the debugging. I’ve had enough for now no more DNG for me!

.JPG vs RAW (.PEF) vs .DNG

arning, I must preface this post with the advice that this is very much my personal view, based on my experience rather that on a calibrated benchmark comparison. I have the choice of using jpeg, two types of propriety RAW  and .DNG as an alternative Raw format on my Pentax K20D. I have found the decision as to which format to standardize on hard to resolve and opted to save both RAW+jpeg where there is the such an option. So here is a little about my (in)decisions and why.

All my current and past digital cameras and cameraphones have all captured .jpeg files and my photo collection is dominated by that format. I have come to appreciate picasa as the very best way to manage jpeg files, its fast, simple and behaves. My many mistress project has lead me to see that previewing jpeg thumbnails is by far my preferred way to cull and organise my photos as I load them. At this stage whilst I might do a little cropping I tend not to do any elaborate post processing. Occasionally at this stage I will email images and very occasionally upload them to the net which generally require resized jpeg version (again picasa does this well) without much photo retouching.

There is also no doubt in my mind that you can coax much more out of RAW files than you can get out of the equivalent jpeg. The reasons for this is buried in the image "development" process happening in the camera, to take the sensor readings and produce an image. There are several steps, demosaicing of the Bayer array, white balance assumption, tone mapping, colour saturation and contrast, then image compression before it is written to the camera memory card.. Probably the biggest limitation is that the image must be packed into an 8 bit representation. Each step requires interpretive decision about how to preform the development and many are not easily recovered if the steps lead to less than desirable processing. That’s where having access to the original information in a RAW format file is a big advantage, but now the steps must be specifically undertaken by the photographer, which takes time and experience. Often a lot of time! The jpeg representation, or at least a suitable small sized thumbnail of it, is also needed at the camera itself to be display on the LCD screen on the back of the camera. 

Pentax JPEG *** (5.9MB)	Pentax JPEG **** (10.0MB)

There is a trade off between the time take to write the large RAW files and the smaller jpeg files, which can mean that you can take fewer images per second, because there can be a bottleneck writing to the card, When you just shoot raw, the jpeg development steps are likely to be run anyway, not only for the image on the LCD screen but most Raw files also include this jpeg thumbnail, for use in quickly previewing the Raw file contents. So in RAW+jpeg both file formats must be written to you to your memory card and it is then just an issue of time taken to write both and extra space required. So fewer unique images fit on the memory card before it is full. Rather than opt for a larger memory card. I've chosen to have several 4 & 8GB cards that have faster read write access, so other than having to change cards more often I probably don't see much difference. There is one exception which is tracking birds in flight and I'll discuss this in a future post.

Pentax RAW  .PEF (13.2MB)	Pentax RAW  .DNG (23.1MB)

I have a Pentax which is one of the few manufacture that offer DNG as an alternative RAW format, Leica and Hasselblad are the only others I know of that can save to DNG in the camera. DNG can have one significant advantage, in my view, the files can be 15 to 20% smaller than the equivalent RAW files if they are lossless compressed. This is may be why many photographers have chosen to convert their RAW files to DNG on import. Anyone that has tried the convert on import into lightroom will know that this is simple but a very time consuming process, On a full card this is not just time for a quick coffee this is tme to have dinner and some. However when I choose that format on my Pentax K20D camera, the DNG files are uncompressed and they are almost double the size of the PEF files. Against this there are some differences, just differences not improvements, in how the images are rendered particularly in lightroom and I can't see any advantage capturing in DNG

Cannon JPEG (4.5MB)	Canon RAW .CR2 (16.8MB)

Because of the time/file size trade-off I have stuck with using more compressed versions of the jpeg files (on my Pentax this is the JEPG *** rather than the uncompressed JPEG**** format) when using the RAW+jpeg. This compressed option is the only one available on my canon when using RAW+jpeg. The images posted above are all jpeg files of reduced size, so they can be dispalyed on the web. So it is not possible to show the subtle details and differences here. I have also uploaded the unedited files to my google+ JPEG vs RAW vs DNG photo album. If you are interested this may given extra insight.

So I hope you understand why I choose to store RAW+jpeg and have opted to stick with the PEF format (and .CR2) rather than use DNG from the camera. I do use the DNG format in some special circumstances but that’s for another post.This discussion is prompted by to my thinking longer term focus

 Thanks again to Jessica Hische for her great drop caps.

Popular Image formats

opular image formats, change with time. At the moment jpeg and raw are the dominant players in storage of photographs from digital cameras. Actually RAW is not a format per se it just means the camera manufacturers format for the complete exposure information has been stored directly from the camera sensor, it is general uncompressed (and therefore large), not enhanced in any way (it will usually look flat on first view) but will have a higher dynamic range and often a higher bit depth than the equivalent Jpeg image. RAW file are often described as digital negatives, which is not truly correct in detail but is a good analogy because, like a film negative a raw file typical require some development or processing before it can be shared on the web (usually after being exported as a jpeg file) or Printed.

Adobe’s DNG (Digital NeGative) is heavily promoted by Adobe as the “standard” RAW format, yet few camera manufactures, especially the bigger ones have not yet embraced it. I have a pentax which does offer DNG as an alternative RAW format, Leica and Hasselblad are the only others I know of that can save to DNG in the camera. DNG does have one significant advantage, in my view, the files can  be 15 to 30% smaller than the equivalent RAW file when it is compressed. So many photographers convert to this format as they upload there photos. Another advantage, which is a two edged sword, is that the metadata and details of how an image is processed can be save in the DNG file, where as no camera metadata and processing details normally have to be stored in a separate file (eg an .xmp side car file) for propriety RAW format. The hidden danger here is you can unintentionally overwrite your original image with new edit and potentially degrade your image in an unrecoverable way.This also makes the DNG file less like a negative and more like a digital print.

JPEG  (.jpg, or .jif) is by far the most popular format for the capture of images on cameras and smart phones and their display on the web. It uses lossy compression, which means some detail is discarded to enable smaller file sizes. The degree of compression is generally selectable and is a trade-off between size and image quality. There are several variation but the common format adopted by camera manufacturers embedded metadata about the camera setting in the file header. It also contains delails of the colour space to be used. Virtually all photographic software and browsers are be able to read and render jpeg files.

PNG was designed as a lossless successor to the gif format, largely because of l;licencing fears over the gif format, and it is probably still the most wisely used lossless format for web display being supported by all browsers. Unlike gif it does not support animation, but it does support extended pelleted colour scheme (eg 324bit RGB & 32bit RGBa). It does not store metedata about the image, which may explain why it have been ignored by camera manufactures Its popularity is mainly in computer graphics and illustration, but can be used for saving photographs to be displayed on the web.

The TIFF format is one of the first image formats, it can use lossless as well as lossy compression and does have higher bit depth for its colour rendition (ie 16bit rather than 8bit). It was for a while the format of choice for very large images and a few camera manufactures did have it available as an optional format. It can be processed by most (but not all) photo management software and is still popular in the graphics, digital illustration areas, as well as the output format of choice from some HDR programs. Most web browser will do a reasonable job rendering tiff files, The underlying tiff standard has nit had a major updated since the early 90’s, but it is another format under the control of adobe.

GIF was once a very popular format to display of images on the net, because of its generally smaller size. It is still popular for some aspects of web page design because it can incorporate transparent zones within the image. a variant of the format also allows animated graphics to be created, and these have become very popular (overly so). If you load 3 to 5 similar images into google+ it will create an autoawesome, which is an animated gif stop motion style animation from those photos. I’m not aware of any cameras that capture photos as gif (some wencams and scanner can). There has been a lot discussed recently on how to pronounce the name of this format I still favour Gif (the sound G sound as in giraffe) but apparently one of the developers of the form prefers Jiff (pronouced with a soft J as in Jill)

Webp is a new format, specifically developed for web display,that is being developed by google but as opensource. It offers significantly small files and faster download. It is available in both lossless and lossy formats to give greater compression removing some detail. It does provide for storage of metadata. It is not yet widely supported but the chrome browser and applications like everpic can display it now. And while on pronunciation this is often called "weppy"

The variation of RAW, DNG, GIF & TIFF are all propriety formats potentially under the control of specific companies, and some worry that if and when these companies fail or probably worse they suddenly enforce strict licensing on patents, then these formats "could" potentially become difficult to use and exchange. I suspect the large number of existing images in these formats will ensure ongoing support.

Finally this is not a comprehensive list of image format, see this wikipedia comparison page to find other formats, but the formats discussed here are probably the most common ones you will come across.

Show which formats do I use?

Thanks again to Jessica Hische for her great drop caps.