Three Weeks On …the cloud photo archive splits into two

              

My cloud photo archiving experiment is continuing but only one of the original contenders, OneDrive, has survived and still running well. I was looking at a way to automated the update process to it and I have gone back to a very “manu-matic” (part manual, part automatic) method utilizing the powers of PhotoMechanic. There is a story behind that choice so I might explain that now.

A real advantage of the OneDrive approach is being able to control the sharing of specific photos with others in a secure (email private perma-link) connection (Dropbox also does this well). Google on the other hand isn’t secured (their idea of secure is just to have complicated file names, but that file name does have a permanent url, which is always publically accessible) AND I’m sure their approach will soon be, if not already being hacked. This private share feature would make such a service a good place for a family photo resource, that is only shared with the “trusted” family members.

This then got me thinking about making the family photos archive separately, and the more I think of it the better this strategy becomes. The difficulty is how to separate out the photos and more importantly the family just wants one or a few good photos not everything. The solution is good rating and culling. Damn, more work! This can be done with lightroom, On1, Aftershot Pro, Darktable using star & colour tab ratings. Providing I have this software automatically writes out. xmp sidecar files, I can make and change these rating at any time. These programs all monitor the .xmp rankings (Pick flag are an exception which none of the software seems to want to exchange, and I don’t use them anyway)  As I am reviewing or processing my photos I consciously mark those photos with the purple colour tab (pressing <Alt>5 in most software, but not lightroom). However I like to recheck my ratings and colours using PhotoMechanic’s contact sheet display view perhaps once a week, I review each days contact sheet (PhotoMechanics Term for the folder/thumbnail view) and I can then filter to see only one star and above rated. I can also check the colour tags and metadata updates if required. and recheck the ratings (eg change * to ** etc.). Next I remove the purple tagged family photos (they will go into a separate archive (see below) and finally I “save as” the selected group to my OneDrive folder on my computer. I am only saving finished jpeg files and have adopted the practice of resizing the photos to 2400 pixels on the longest dimension and jpeg quality to high. This approach to reduces the file size but in future being able to print approx 10” by 8”.  Finally it wpould be possible still be able to upscale because I have not utilized the .jpeg lossy compression to reduce size.

There was a secondary benefit from reaching back to a manu-matic step, after uploading my better photos into the cloud archive, without family photos included. I could switch the filtering to only include family photos (the purple colour tag and 2 starts of greater) and upload those to my trials of a private flickr family group and revised dropbox family folder. More is likely to unfold with this second type of cloud archive, so say tuned for updated.

Ransomware & other ways to loose your photo collection

The Wanna.cry hullaballoo brings to light the vulnerability of keeping your photo collection on-line, on a computer or phone but in one place. This is running a big risk, do something about it now.

Avoiding most Ransomware, is not difficult, its just needs you to make small changes in normal behavior and regularly be wear of or do a few things. Be careful about clicking on links in unsolicited emails and phone message that have the characteristics of a phishing exploit. Be aware of identity fraud and becoming a little more private, be careful and aware in what you share. Even basic snapshots can hide a lot of useful information to a stalker (eg location) or hacker. A good example is the common trend for people to post their boarding passes from the airport gate on Instagram or Facebook which gave hackers enough personal information to gain free entry into accumulated frequent flyer points which they immediately stole. The most important protection is keeping your security/protection measures up to date. Which means installing rather than ignoring those security updates. The final and most important way to not be caught out with ransom wear is to keep an up to date backup, under your control, then the ransom on on a web site/computer becomes meaningless.

However ransomware, virus or worms are not your only risks, Equipment loss damage of malfunction is a much higher possibility and in some cases approaching a certainty. In the case of mechanical spinning harddisk it is a certainty that they will eventually fail (fortunately generally their life will be about 20 plus years). Equipment particular smaller portable items are also prone to get damaged or lost. Mobile phone really don’t like getting wet or dropped. There contents sometimes survive, particularly if have them on a microSD card but I hear many sad stories  of photos lost this way. Making sure your phones get backed up properly is the guaranteed way to avoid the risk. Sure you can sometimes recover files from damage harddrives and phones but it can be very expensive, its much easily to go get new equipment (your going top have to spend that money anyway and pick up an reload from your backup.

 

So what is proper backup. There is lots of advice on the net but the best approach is the 3-2-1 method (see the basics in the David Bergman Video above). Have three copies (yes three) on two media/devices (eg disk, DVD etc) and one copy off-site (eg on-line, bank vault, a friend house across town). There is a lot of options in how you actual do these steps and set up something you can easily manage. It may involve some extra expense (like buying a second hard drive subscribing to an on-line backup service) but remember you are responsible and the should take control of the steps (ie don’t expect a free social media service to do anything special about “your” photos.)

Go start that 3-2-1 backup today, don’t delay it could be too late!

Backing up the actual photo files is also only part of the issue. There are important aspects like how you rate the images and organize them, and in the case of non-destructive editors how the image has been manipulated. Some of this information can be written back into the header of Jpeg files but most photo processing software today stores this separate to the software. In the case of lightroom it is all stored in database  its calls the catalogue. other software such as picasa store this data in a special ini file in each directory containing photos. The original adobe bridge software stored the information in a readable ascii companion file (.xmp) normally known as a sidecar file. Whilst these are not standard, and adobe itself has muddied the waters by changing some format so you need to know the ,they have been around a long while and many other software packages can at least read the metadata and ratings in these files. Some packages (like Corel’s Aftershot Pro and XnView) can also write the basic .xmp files. You need to look at the help (or manual) for the software you used and work out where these files are and how copies can be included in your normal back up set. While Lightroom does itself keep a separate backup of its catalogue it is likely to be on the same computer as the software and photo library. So it is important to include this in the 3-2-1 steps.

If you’re a regulkar reader of this blog, you will know I keep harping on the issue that backups are different to archives, but while they are also copies they should be organized in such a way that photos can be easily found (that metadata discussed above) and there is also the even bigger issue of what format have the best longevity.

Backup your software and operating system by creating a recovery disk (eg in Windows 10, you get the opportunity to create some recover media from which you can reboot your computer) in the case of hardware or software problems. Its fairly large so you will probably need a 16 GB USB drive. I made mine (for my three windows 10 computer) in fluro green) so they look different to any other USB memory key I have. Luck I made the effort because my main desktop reported an update error, and after clearing the upload and reloading the security patched it still seemed unhappy and after the next shutdown it refused to start. Rather than panic I got out the recovery key and rebooted from the USB. Five minutes later I had my desktop back. So make that recover media now. Another item you must remember to backup is the software you use to view your photos plus the registration Keys that you got when you purchased the software. Don’t assumed that you will be able to always download older version of the software. I know corel, adobe and microsoft are not making what they consider superseded available for download. So Search out that install package and include it in your photo backup. Digital photos are binary files and need to be interpreted and rendered to suit your output device, by software. The big risk with proprietary RAW photo format (which many on-line photographers encourage you to use) is they may not be supported in future software. Jpeg is an accepted public standard and so ubiquitous that it is unlike to not be viewable but .DNG, .CR2 .Nef etc might be a little like having a beta Max tape but no player.

Regularly make your best/favorite/precious photos in photobooks, or at least 4” by 6” (10 by 15cm) prints. This is really a physical variation on the backup (in another media) but it does work. Think of all those old shoe boxes with photos from your parents, that have survived the years. Unfortunately many get trashed in post death cleanups. They are controlled by you but can still be more directly/privately shared by you to just those you want to see them.

“Privacy is becoming a privilege” … Paul Thurrott (Windows weekly podcast)

Not all is harmony in the XMP World

Whilst many software applications can share metadata via XMP files, the situation is far from a recognized standard.  XMP is an extensible XML style format, which means it can be customised.  Applications that read this data, need not be able to understand everything.  They just need to honour (and not modified) any such data.  Adobe itself use the XMP the file to store details of proprietary Post Processing methods within Lightroom, These methods are unlikely to be known by the software, and that software should not be expected to reproduced the exact Lightroom process.  Corel’s Aftershot Pro also uses its own proprietary XMP format for its processing, but also offers the opportunity to save standard XMP files by which it means just the metadata.

Is in the area of metadata that there does seem to be good agreement between packages.  Specifically key wording, star ranking and the colour flags/patches or classifications do seem to be reliably exchanged between programmes.  Most of the EXIF data seems to be so exchanged, when exception might be camera settings and lens identifies (usually of read from RAW files) that don't always make it into XMP files. Some some software does not acknowledge IPTC metadata.

Finally I don't expect to be told what isn't read from and XMP file.  Anything that isn't under stored get ignored.  So for example you may have set the Likes and Dislikes in OnOne's perfect photo suite, but they will not be interpreted as Flag as Pick or Set to Reject  in lightroom so they just don’t show up.

Taming the shrew

I have been carefully investigating a way that lets lightroom co-exist nicely with other packages across a range of computers. Specifically I am trying to avoid having to commit to let lightroom force a specific workflow on how I “manage” my photos. I much prefer to be able to browse photos where ever they are. Yet lightroom’s development modules can be extremely useful (at times) so I am keen to let her free but at the same time avoid all her high maintenance demands.

My solution has two parts.\

1) I have decided to only setup the master lightroom catalogues in one place.  Yes I have multiple catalogs.  I've come to the conclusion that I have too many photos to have a single catalogue. The performance of the library module in Lightroom definitely takes a dive passed around half a million photos.  My solution has been to have one catalogue for each year. These may be a bit small but they are easy to manage. I only set up these catalogues on my archive and the catalogues are on the computer it is connected to.  Elsewhere lightroom is only used on the understanding it is temporarily and any cataloguing is only temporary, but with the specific condition that an .xmp file is always generated for any activity.

2) .xmp side car files to the rescue

One of the aspects that led me to like Picasa straight away was the ability to cull, rate and keyword photos as I preview them immediately after loading. At that stage I was only shooting jpeg format and this information was immediately incorporated into the embedded EXIF data within the photo. However this embedding only works in jpeg format, not the various RAW format. Thus this extra data the metadata followed the photo wherever it was stored. However in the default lightroom settings this information is stored in the central catalogue, if the photos is moved or copied by any program or utility other than light room this information will not follow the photo. There is however a mechanism in the Adobe photo management philosophy that can store this information for each photograph separately. Instead of including it in the EXIF data embedded in the photo itself it creates a separate small file, with the extension .xmp (for Extensible Metadata Platform). This is actually an old fashioned feature, probably with origins in the Adobe bridge software. It means you must copy two files, the original photo and the matching xmp file together. Finding the way to write this magic XMP file, is not immediately obvious in the lightroom menus. It can not be found under the more logical metadata menu, rather it is somewhat buried under the Edit/Preferences  and select the General Tab and the press the Go To Catalogue Settings button on the bottom right. Finally select the metadata Tab on the far right and tick the third item Automatically write changes to XMP.

Whilst Picasa does use XMP style metadata for its facial recognition face tags, it cannot read or write a standard metadata as separate adobe xmp files. However both OnOne’s Perfect Photo suite 9 and Corel’s Aftershot Pro can read and write the metadata into and from .xmp files, since both of these have browse modes I can also easily to the rating and keywords as a preview photos and this will find its way with the photos, wherever they are and even be loaded into lightroom when it imports of photos of my archive catalogue.

A Sting in the Tail

The separation of the post processing from the original photo files (be it jpeg or Raw) in the so called “non destructive” editors (like lightroom, aperture, aftershot pro and picasa) brings with it a lot of power because you can then go back and undo and change the post processing steps later. However there is a sting in this approach’s tail and many aperture users are probably going to realised very soon. Most of the “non destructive” work is carried out by proprietary software and stored as procedures in either databases or specific files. Whilst the photos themselves may be moved around the broader internet, cloud services and computers, even smartphones and tablets, moving this post processing along with the originals is generally not possible. There are a myriad of reason for this incompatibility of proprietary approaches, some social services even stripping metadata, bandwidth and file size limitations.. Further the trend within the big software developers is definitely towards serving the smartphone and tablet market and storing photos in the cloud, but this gets you into an even deeper propriety mess, cloud services are very definitely little closed silos, and sharing between of across them a challenge. The situation is a hodge podge and crying out for some standards to be developed for exchanging at least the most basic processing options, such as colour and tonal adjustments, cropping and straightening.

I’m not sure there is a clear solution yet, although at the moment you can save your ”cooked” images as either jpeg or dng (but dng is actually a propriety format). The big disadvantage with jpeg is that its precision is size related (a file that displays well on a smartphone screen may not be adequate to print, and a lot of information can be lost compared with the original file). Dng is being promoted heavily as a great standard by adobe but really its acceptance is at best lukewarm.. The other good practise at the moment is to write the small side car files (.xmp format, again another adobe proprietary format) which can be exchanged between several programs, at least for the metadata and often the rating of photos. The highly prompted Ipad app version of lightroom, can render the images according to the post processing on the computer but can not undertake most of this post processing a fresh, but it will allow you to rate the photos and update some of the metadata. Lets hope the aperture demise and user dissatisfaction does drive the move to more standards to exchange metadata, rating and at least basic post processing option.