Refining my "Air Gapped" Backup/Archive, rather than a New Year’s Resolution

I’ve not been so diligent; my "air gapped" archive/backup system has been sitting on the shelf for the past two years, gathering dust. You know how it goes. I had good intentions about checking those old hard drives every six months (they need to be "exercised" to stay healthy), but eye issues and life got in the way.

When I finally fired it up again, I accidentally left the Wi-Fi on, and Ubuntu immediately reminded me about updates. At first, I'd been avoiding updates thinking "I barely use any features, why bother?" But now seemed like the perfect time to upgrade and add some new tasks into the system.

Enhancing the 3-2-1 Backup Strategy

This whole approach really came out of the ransomware era (still very much a security problem). The idea is simple: keep an offline backup so you can recover if something goes wrong, and make sure you're actually verifying that your stored data is correct zero errors.

 Just looking at a photo or document isn't enough to know if it's been corrupted or tampered with. How do you really check a file? It depends on the type of file and format, and there’s a growing miriad of different ones to worry about.

My Practical Approach

For photos, I'm using MD5 checksums, basically digital fingerprints for each file. Adobe has its own proprietary checksum system in Lightroom and new content credential tools, but I'd rather stick with the commonly used standard.

Raw files are trickier. They vary between camera manufacturers and have already changed over time and camera models.. Some photographers convert everything to lossless TIFF files (which can become huge), others swear by Adobe's DNG format, but I'm not seeing widespread adoption outside the Adobe ecosystem.

Video files are a more difficult area, they rapidly gobble up space and I've got formats from software I have no idea about. For now I will just have to stay with VLC video viewer, which seems to cover most video formats.

The Reality Check

I'm being realistic here, I’m getting older, and I have half a million photos. Nobody wants to wade through all of that when I'm gone. So instead of keeping everything organised just by date and time, I need to actually curate the mess and my legacy. Pull out the family photos, separate events and places, get rid of the junk. I'm planning to explore best ways to identify the high-value stuff worth preserving properly.

The plan is to do this reorganisation on my main computers first, then pass over a well-organised set of archives to the air-gapped system. It's a bit scary because I'll be deleting files, so I'm keeping the full backup for at least a year as a safety net. Currently sitting at 5.5 terabytes and hoping to cut my archive at least in half, but we'll see how realistic that is, same time next year.

Photo Ingestion: From Chaos thru Workflow and back to Ad-Hoc

Over the years, my photo ingestion process, jargon in digital photo world for uploading your photos say to a photo managment system, has evolved from simple to semi-chaotic, shaped by changing gear and software quirks. I started out loading JPEGs straight from the SD card using camera-branded software, then shifted to Lightroom Classic’s folder-based system. It was slow, and I quickly ditched metadata tagging in favour of a manual copying of the files and folder structure, yearly, monthly, daily, based on the camera’s default naming.

As RAW formats became standard and file sizes ballooned, I had to juggle multiple SD cards and software that didn’t always play nice. Initially my Canon EOS 1100D required its own viewer for RAW and MOV files. Then in 2017, I won a copy of Photo Mechanic, a game changer. It mimicked old-school contact sheets, let me tag and sort on the fly, and even added GPS data, which was perfect for my sun-chasing road trips around Australia. I also had two Pentax DLSR cameras, a K200 and a bigger K20, both nice cameras and I really liked Photo Mechanic. It was a very useful little programme when loading photos particularly when you’re travelling because it significantly reduced the amount of time and fiddling.

I had become interested in mirrorless cameras and got the little OMD-10. Unfortunately, Photo Mechanic didn’t support Olympus RAW, so I pivoted to Olympus, now OM Workspace. It was free, handled most formats, and even updated camera firmware. Slowly, it became my go-to for the OMD-10 and later the OMD-5, which became my main camera.

Now, with just a single smartphone and mainly Olympus gear, I can upload in a mixture of USB cables, card reader, Wi-Fi transfers, and Microsoft’s Phone Link. It works, but it’s again ad hoc. 

The challenge now is keeping everything backed up and organized across formats, folders, and devices.

Preserving Albums, Collections & Organization as you move digital photos

In the process of downloading my files from flickr I’m reminded of the unexpected issue most folk find when trying to transfer a lot of files between different photo managers. Things like EXIF and key words can be embedded in the files (in the case of Jpeg and Tiff, also DNG but not other RAW formats) and sidecar files (eg .xml can also carry this data with photos as they are transferred between packages. Unfortunately the organization features such as Albums (or Collections in Lightroom) are not transfered and/or not compatible between packages.

There is a simple solutions I have written about before. It is using special key words to identify the albums. I use names beginning with the tilde key ~ and write that into the keywords (or tags in flickr jargon). In most software you need to enable writing back of the keywords into the embedded data and/or writing the sidecar file beside each file. When you load into the new software it will read in these keywords and you can selective filter on a given ~keyword then do whatever is required to create the new album or collection.

Very simple really.

Transferring a Picasa based photo library spread across a couple of computers

There will come a time with any digital collection of photos that they need to be transferred to a new computer, perhaps a different operating system. If everything is well organised in one directory and subfolders, the whole set can be copied to an external USB harddrive and carried to the new computer and the copied onto it. Pretty simple! However if you check  you might be surprised, what has happened to your previous edits, the tagging, star rating and album organization (collections for lightroom only users), they are much more difficult to ensure make that leap across computers. Now add the fact that not all photo collections are perfectly organized and often the photos are spread across several drives of across the local area network. Transferring photos might just become a nightmare!

I rediscovered that picasa is still a wonderful tool to manage your photos (well jpeg libraries to be specific). While google abandoned picasa a few years ago, but you can still download it free from several popular download sites (just make sure the site is trustworth and doesn't come with unwanted blotware and especially viruses, worms or ransomware). In later version (approx. 2015 onwards) the picasa web component was changed to use google photos (and I strongly recommend not using the autobackup to the web feature).  Google wants everyone to use their google photos system on the web, not do anything on your own computer which does not suit me and I assume will not really suit most people.

“For those who have already downloaded it, it will continue to work as it does today. But we will not be developing it further, and there will be no future updates.”   ….google post on picasa user web forum

If you already have Picasa,  you will know it has some good photo editing tools (its a pity snapseed never made it into the collection), is very fast to load photos, incredibly easy to understand and probably still has one of the best facial recognition features. It is not so great at enhancing RAW format photos (it can display and manage them). Further the “copy the photo directory” approach works well for picasa because it has a usually hidden file “picasa.ini” in each directory which stores the processes such as cropping etc. and basic organization, as well a simple start rating on each photo you have editing. When the the entire directory is copied onto the new computer and picasa reloaded it can read this file and render the photo as it was on the old computer. I discovered some time ago that this did not copy the facial recognition organization, but there was a complex workaround involving copying the hidden picasa database. I was pleasantly surprised to find the Tools/Back up picture feature in Picasa 3.9 (the last downloadable version) will also copy this extra information. I was using this backup approach for a different reason, I knew it would search out the photos from where ever they are stored and put them into one big set (generally on an external USB drive for easy transfer).

The approach is very simple and easy to follow but I figured I should prepare a simple step by step through the process anyway and post here, because I suspect A lot of people will want to do this as they ungrade computers over the next few years. You will need picasa 3.9 to get all the facial recognition transfered (otherwise see that workaround above).  Get an external USB hard disk and plug it in (make a note of the drive letter it is given). Or alternatively you can find somewhere on your local area network with lots of space (a NAS drive would be perfect albeit slow) Now Run Picasa on your old computer and using the tools menu find the back up picture option.

This will bring up the backup dialogue panels on the bottom of the screen. The default is normally set to somewhere on your hard drive, so you will need to  select the New Set option. Then enter the address of the USB drive (or network location). I’ve put a 1 2 3 marks on things you need to address on the first panel to get your back set defined. Firstly give it a name, Secondly select on the Disk-to-disk backup type and using the choose button locate your destination. I had already created a sub-directory on my USB drive F: called transfer. Finally you need to define which files to backup. Picasa was never used to manage movie on this computer so I excluded them (to save space). If your picasa library is full of screen captures and or downloaded web graphics and you only want your original camera files you can try the last option. Otherwise it may be safest to select All file types to backup. On the second panel you can set up selective folders to backup but when transfering computers just click on the select all button. The final step to start the backup is to press the Burn button on the far right hand side (this term comes from the days when you “burnt” CDs, which you still can do by the way) This process can take a while and there are no progress bars but a pop up screen does tell when it is complete.

Final take the USB drive (after using the safely remove and eject option in your operating system) over to your new computer. If you look in the directory created (transfer in my example) you will see there is a collection of files and sub-directories and one of the files should be PicasaRestore.exe. Click on this to run it.  You can change the location of the restored files but most often the select to restore to original locations is fine for a an inter-computer transfer. Again this can take some time but you can see progress details of the files copied and setup steps and also another pop up when the transfer is complete.

When you start Picasa on the new computer it may take a some time in background to scan through the library and connect up all the faces and some organizational aspects. However you should find that your complete picasa library is fully and safely transferred and available on your new computer.

Looking at XnView MP as a Picasa Alternative

I have had XnView on my USB Darkroom Key (its always in my camera bag and has portable Apps so that if and when I only have access to a cyber cafe or library computer I can still work on my photos). It is a really simple to use but feature rich photo viewer/manager, I have also had XnViewer MP (a multi platform enhanced version) running on a repurposed Linux machine and my desktop computer and they do get frequently used, However picasa is still my go to application for loading organizing and culling new photos. The newest version of XnViewer MP (0.81) may change my mind.
There are three reasons

1. Its RAW rendering is fast and decent. I would not suggest that it is a fantastic replacement for other RAW editor but it is is an ok conversion. The downside is it converts the RAW image into an 8 bit jpeg, and thus you will loose a some of the potential dynamic range,
2. It now includes (from version 0.80) the ability to read and write metadata in .xmp sidecar files. Which means you can transfer rating and metadata into other photo mangers (that can read the adobe sidecar format). You have to go into settings under the browser/metadata tab and click on the options highlighted on the right. Again there is a small caveat in that XnView does not include a pick and reject flag (so useful in lighroom and other software) when first reviewing and culling newly loaded photos. Also the XMP compatibility does not extend to post processing options (just the metadata and ratings).I can not write to xmp in picasa so this is a big plus.
3. It has a simple tools/import and sort feature that can load photo from a variety of sources (the can be camera, card readers, external drive, they just need to be connected to the computer and have a drive letter and/or folder directory. Whilst XnView MP does build a catalogue entry and thumbnail at this time it is way faster than lightroom for example. You can also add metadata and do some filtering by time (assuming the photo has EXIF data). It can also write data to a subdirectory yyyy_mm_dd according to the day it was photographed and this is how I like to organize my files

There are heaps of other well loved features that have come from the original XnView program.  The real power of XnView is the organizing your images, doing a few basic edit and sharing your photos via email or getting them to go to a social media site (nothing ”one click” automatically upload to those sites but there are the right tools to make the files the appropriate size and format). There is also the great range of batch editing tools which can simplify task like changing timestamps to synchronize camera times and external GPS tracks, for geocoding (an all to tedious a task in lightroom).

There is also the ability to read and write an extensive array (the web site claims 500 format) of graphic file formats. Ok there are only a few formats I regularly need but it is handy to be able to at least look at the strange files that turn up from time to time in graphic art. Thus this would make a great for managing the reference photos for a graphic artist who normally works in illustrator or Corel Draw/Painter.

I’m still only luke warm on it being able to replace picasa in my toolkit, but I have made a commitment to put this version on all my computers along side picasa and give it a fair work out.

One disappointment already is on the HP Spectre  (2 in 1 windows 10 laptop) where it its decidedly awkward to use in tablet mode and where it can be very touch screen frustrating.

Hello from my new photographic travel mate

After the frustration of travelling with the upgraded laptop, that was having trouble connecting to wifi, amongst other frustrations. I felt it was time to get a much lighter but still powerful enough PC to use specifically for my photographic endeavors as I travel. The basics were pretty simple to map out, I wanted a browser, email, something run to Lightroom and OnOne 10 and Aftershot Pro. That would require 8GB as the minimum workable memory, an SSD drive if possible at least 256GB, a decent screen and light weight. Whilst there are lots of laptops that fit the bill I was attracted to just 3. The Microsoft Surface 4 Pro, the HP Spectre and the Apple MacBook Pro, Whilst the Apple with retina had the nicest screen it was by far the most expensive and I would have to restock my software tools and I’m not a fan of apple OS anyway.  Relearning the way I work while I travelled was really not an option. The decision between the Surface Pro and the Spectre was more difficult. They are both 2 in 1 machines that can run as a Windows 10 PC but also be run in tablet mode and have touch screens. It ended up just being the spectre is a sexy build, very James Bond, metal case and much nicer keyboard and the backlight IPS screen has a lower resolution than the surface but looked better with my photos (I spent some time on each computer looking at my recent daily photos on flickr, then stress tested the wifi access and checked out their keyboard. Not exactly a detailed technical investigation). So all three would be fine but the spectre looks more durable & stylish, something I could confidently lug around and still get down to serious photographic post processing and management. I'm not used to touch screens (yet) but the spectre comes with a pen that works a lot like the pen stylus on a wacom tablet, so that feels natural and powerful for me. Keyboard and pen are extra costs on the surface putting it significantly higher in price for a comparable setup. At 1.4 kg with the power block its less than a fifth the weight of my toshiba laptop and power brick (7.5kg). Travel fun awaits...

Shredding Software at a great rate

To be able to access the internet I have had to set up new accounts on my limp-a-long windows 10 laptop. Some software I had to reinstalled but the photo applications I am using OnOne, Corel After Shot, Picasa And lightroom kept working. However now OnOne & Aftershot keep wanting me to re-register. Oneone has now even stopped working the trial periods is apparently over. It must be just 6 days since that is when I set up the new account.  I have purchase both package and install them on this very same computer months ago (back when it was a well behaved Windows 7 PC). Ok under a different account name. So I went back to the original account (which can not access the Wifi system here on Norfolk and while Aftershot pro seems ok, One One is still refusing to work and now lightroom is having tantrums as well disappearing into silence. So I have a very sick windows 10 upgrade on a decent computer gobbling up the software applications I want to use. NOT HAPPY

At least Picasa and Paint.net have not deserted me (yet!)

To rub salt into the wound Dropbox has just sent me a message that it is not able to synchronize.

And of course I don’t travel with the registration keys and installation packages for all my software so I’m feeling very abandoned,

Thinking about PhotoFlow, rather than WorkFlow

It would seem everything on the Net that is photography oriented will at some time try to sell the perfect workflow. I’m not a fan because these formalised workflows soon become constraints that limit what you can do. In other words I see a rigorous workflow as anti-creativity.
I much prefer the idea the physical flow of the photo  has a lot more to do with being able to work flexibility. (I know it is not truly physical but a spatial workspace is a good analogy). For start I have several mistresses (photosoftware  like Picasa, Lightroom, AfterShot & OnOne) on all of my computers and they don’t exactly play together well. Next I am progressively changing how I organise my photos. There is still an archive of everything (less the photos I cull on import). I now have four places (separate folders on network drives) where I save and work on photos, The most important is Portfolio which is the place I store my best photos, they are not necessarily shared on the net. I have become increasing convinced you need to save your favourite photos in their fully processed view. The three other grouping are PhotoScrapbook, VideoScrapbook & Family are more work places and photos can come and go. If these get post processed the sidecar gets sent to the archive. In all these workspaces I use albums to group the photos (Rather than folder location on the disk).   Finally I have two different ways I share via the Net, the obvious social network places (to open up the image to anyone), and more importantly “private” sharing (where I can select those to share with and only them).
My workflow is still essentially a simple Shot, Edit & Share but I also see it as simple steps to move working photos into a suitable workspace.

Where are we going?

For many years the emphasis in digital photography revolve around the work flow. It began as a way to simplify the learning of complex software like photoshop and lightroom on a computer, but softened into a more generic shoot, edit and share. At roughly the same time social media became big and what drew a lot of views was those digital photos being shared. These two themes dominated developments in the photocentric internet world for most of the past decade. Then cloud computing raised its head and after a few train wrecks, and then the voracious gobbling up of promising startups by the social media and bigger computing and/or internet groups. The theme for this years is definitely one place to store all your photos. I suspect all the contenders are aiming for that sweat spot at the intersection of all these themes. Yet from my perspective all of these offerings have at least one shortcoming which disappoints, since most are essentially free perhaps complaining is not being reasonable. However my photos are mine and I want to look after them. I want to do a range of tasks and I want to be confident that the images are only shared when and where I want. I think it is time to leave the work flow, cloud and social media constraints and refocus on tasks that need to be done. These are methods and skills, some equipment/ hardware, some software. Using these processes and tools that can be learned and mastered in the manner of any true artisan. Then again I might be in the minority, I cringe to think most folk are happy to push the shutter button, click on autofix or the make zany filter and flood the world with wasted bandwidth.

And the winners is...

Whilst Google photos and flickr are slogging their way along, vying to be first to upload any image from my phone, I realised that there are already two clear winners in this race and its not the tortoise or the hare. First is Dropbox, which I have reinstated (I suspect it might have been nobbled by either google backup or flickr's uploadr but it did have a significant app update around the same time, so I haven't notified the stewards yet!). I didn't find carousel so useful but dropbox is really simple, solid and still has the very logical private sharing options. I could live without it but I love the reliability of knowing soon after I find a WiFi spot my phone images are up on a private cloud (flickr could safely substitute here). The second is a surprise, its QuickPic, an alternative to the native gallery app or Google Photos itself. Compared with Google Photos its a real lightweight with a tiny memory footprint but its fast and does exactly what I expect from a photo gallery app. More importantly it plays nicely with other phone apps.

Whilst the other race is still on, flickr is becoming my favourite as a stayer.

Red Sky in the morning

I must admit I’m normally still snuggling in a warm bed at this time of the morning, and it has been a very cold spell. However I was up and awake early today and able to capture the first light of dawn. Giving me a great opportunity to compare my “girls” (the three current mistresses, aka photo managers/editors, I’m using rather than picasa/google photos) as I would normally edit/tweak them. The Photo on the right is the original unedited Jpeg. The photos below are edited by AfterShot , Lightroom  and Perfect Photo Suite 9. These three systems render my .pef file (Pentax’s RAW format) differently but their RAW file renders all look very flat compared with the jpeg. I liked the dark silhouette of the trees, so I have left the images very low key and just just concentrated of adding a little extra contrast and vibrance.I also straighten and cropped each version. In the edge they all look somewhat similar, supporting the observation that it probably doesn’t matter a lot which software you use as long as it does what you want.

AfterShot Pro 2, which has become my first stop on my laptop. In this case I have just click on Perfect Clear and lifted the vibrance slider a little. With a straightening and crop to finish. All over in seconds
Lightroom 4.4, was a little more considered and all done in the development module, lifting clarity (rather than contrast) and a little vibrance (not saturation) then straightening and crop.

Perfect Photo Suite 9, was a actually my second port of call but I have left it till last as it seems to match my memory of the moment best. I started in Perfect enhanced (which is where cropping and straightening are normally undertaken. Whilst there I couldn’t resist using their wonderful content aware magic  perfect eraser to remove the two light poles (which would have bee a very time consuming task with a cloning tool on the second post behind the tree). I also finished with perfect effects, a natural Dynamic Contrast, A fall enhancer Colour Enhancer bought back to 60% layer opacity and a soft darker vignette

not so fast now

Neither of the two runners are drawing away, in the smartphone photo upload to the cloud race, while I am in low connection - infrequent WiFi land. Not that that worries me, I’m used to uploading photos from my phone regularly. A few images are getting through to the separate clouds, but interestingly they are not the same photos and appear at different times. I haven’t counted exactly but google is probably ahead

The biggest issue I am having is trying to group the one’s I want not those chosen by the google photo assistant (aka annoy-ant). Put simply I don’t like google photo not allowing me to use those new potentially good tools and wasting my time with photo I’d rather overlook. Changing the name from autoawesome to assistant doesn't help, it just annoys me now. Also there is the confusion of terminology now that collections are being used instead of album in some places but not in others, For example if you click on a photo on-line it says add to album but the default view is via collections (aka albums). I also ran into a catch 22 when trying to create a new album from a selected photo when I click on add to album it just showed a dialogue of selected album thumbnails to click on, I could not find anywhere to create a new album of collection on the web version of google photos, BUT it appears you need to be on-line to do it. I had to resort to using the old reliable picasa to create a new album. I can add a collection on the phone and it becomes an album! Its all a bit messy and to be honest I just still prefer to do my organizing in picasa on my laptop

The second issue is how to share the images privately with just nominated family and friends. I have for a long time used picasa web albums (and later google+ albums) to share a progressive updates to an album dedicated to my travels. However some time last year this facility was changes so that the photo could only shared with those that had a google account (which annoyed a lot of people to the point where they would bother looking). Has google photos fixed this? It claims it has. You can get a link you can share and it seems to work. However it is not clear if the collection/album remain private. This combined with the restrictions on staying organized if I’m not spending a lot of time on-line, has made it a bit to difficult to set up my normal progressive album/slideshow just for family. I am still preparing an album in picasa of my travels at the moment but…,. I’m frustrated by try to understand what’s going on with the sharing

This is where flickr is streaking ahead, it is really clear what is private and what is public (via a sharing icon watermark) and when you share private photos you get a link and it clearly states that the photo can only be seen by those that receive the link. If you email this link to a family member only they can see the photo (which is the way it should be) Once again you need to be on-line (ie connected via 3/4G data or WiFi) before you can do any organizing of your uploaded photos in the flickr app. However the phone app and web features are much closer to each other, so you get an integrated feel and they do the job simply and well. I’m actually changing over to using flickr for my progressive trip album to share with family & friends

And the race is on

The new apps on my phone are all busy trying to upload every image that I collect, soon after I photograph or collect it.  At the same time drop box, upon which had to began to rely to upload new phone photos whenever I have WiFi access, seems to be falling behind.

The new andoid google photos app seems to have infiltrated several other apps on my phone, including the native android camera and gallery apps.  Despite the fact I had turned off its predecessor desire to upload, it is happily vacuuming up any new graphic images and posting them to google photos whenever a WiFi connection is available (at least it remembered this part of my settings). Otherwise it seems harmless.

The new android flickr app has also taken upon itself to upload photos from a phone.  However it is probably more polite in doing this.  It seems to only upload from a limited number of directories, specifically once used by my camera apps.  Further doesn't rush to upload new photos, I have the feeling it trickles them up.  Only when I am specifically looking at new photos in the flickr app does it upload things.

So in the race to grab all of my images it appears to be a rabbit and tortoise affair.  I like fables and therefore have decided to let the race continue for a month or so, providing I can see its not going to end in tears.  The vast majority of my phone photos are unimportant anyway so I don't want to flood my “social media” presences with such mundan-ity.  So fingers crossed and let's see where this goes.

There is a big difference in what I see on-line of this race to collect everything.  The tortoise may be doing better here.  All the photos are being flooded up into the clouds are private, which is good.  Many are being automatically tagged with the most common subject tags, also good.  They are displayed in my camera roll in date order, importantly with the small padlock icon to indicate that they are private, very good.  It is a relatively simple matter to click on a specific photo and the click on the two arrows icon in the upper right to display the photo page. Then under the heading additional info the first line is viewing this photo and you are able to select from appalled and list from private through family and friends to public, for public photos you will see the padlock icon change to an eye, beside which will be displayed the number of views of the photo.

In comparison the rabbit, the google photos web view is a bit more brash. The icon are larger, the layout is still a time line (rather than albums). What is missing is any indication of weather the photos are private or public. In fact I can’t find anywhere in the google photos view that can assure me they are private. Not good enough! I’m pretty sure they are private and will remain so until you share them, but I can not confirm that at the moment So whose the bunny here?

Taming the shrew

I have been carefully investigating a way that lets lightroom co-exist nicely with other packages across a range of computers. Specifically I am trying to avoid having to commit to let lightroom force a specific workflow on how I “manage” my photos. I much prefer to be able to browse photos where ever they are. Yet lightroom’s development modules can be extremely useful (at times) so I am keen to let her free but at the same time avoid all her high maintenance demands.

My solution has two parts.\

1) I have decided to only setup the master lightroom catalogues in one place.  Yes I have multiple catalogs.  I've come to the conclusion that I have too many photos to have a single catalogue. The performance of the library module in Lightroom definitely takes a dive passed around half a million photos.  My solution has been to have one catalogue for each year. These may be a bit small but they are easy to manage. I only set up these catalogues on my archive and the catalogues are on the computer it is connected to.  Elsewhere lightroom is only used on the understanding it is temporarily and any cataloguing is only temporary, but with the specific condition that an .xmp file is always generated for any activity.

2) .xmp side car files to the rescue

One of the aspects that led me to like Picasa straight away was the ability to cull, rate and keyword photos as I preview them immediately after loading. At that stage I was only shooting jpeg format and this information was immediately incorporated into the embedded EXIF data within the photo. However this embedding only works in jpeg format, not the various RAW format. Thus this extra data the metadata followed the photo wherever it was stored. However in the default lightroom settings this information is stored in the central catalogue, if the photos is moved or copied by any program or utility other than light room this information will not follow the photo. There is however a mechanism in the Adobe photo management philosophy that can store this information for each photograph separately. Instead of including it in the EXIF data embedded in the photo itself it creates a separate small file, with the extension .xmp (for Extensible Metadata Platform). This is actually an old fashioned feature, probably with origins in the Adobe bridge software. It means you must copy two files, the original photo and the matching xmp file together. Finding the way to write this magic XMP file, is not immediately obvious in the lightroom menus. It can not be found under the more logical metadata menu, rather it is somewhat buried under the Edit/Preferences  and select the General Tab and the press the Go To Catalogue Settings button on the bottom right. Finally select the metadata Tab on the far right and tick the third item Automatically write changes to XMP.

Whilst Picasa does use XMP style metadata for its facial recognition face tags, it cannot read or write a standard metadata as separate adobe xmp files. However both OnOne’s Perfect Photo suite 9 and Corel’s Aftershot Pro can read and write the metadata into and from .xmp files, since both of these have browse modes I can also easily to the rating and keywords as a preview photos and this will find its way with the photos, wherever they are and even be loaded into lightroom when it imports of photos of my archive catalogue.

Automatic Grouping of your photos under Google Photos

There are some very good aspects of the automatic organising of your images because Google Photos goes way beyond just checking embedded geocoding or even metadata. The technology genuinely uses an artificially intelligence approach to image recognition and can identify things and places. However it is not awesome (yet!). It gets a lot of the photo content right, but for my photos there are more misclassification than I would like (see notes below) Also I have not seen the people (face recognition) come up despite the fact that I have been using facial recognition in Picasa for several years and many of those photos are in my on-line google photos.

There are a few obvious misclassifications in the grouping that goggle plus photos finds on your behalf. In particular for me it has taken images of the moon I taken of the full moon, and these had been placed in three groups. Firstly of two place I have not been to (and I have been to a serious lot of places) Segovia and Malaga. The interesting thing about the Segovia grouping is when I clink on that grouping it reports no results! The Malaga group did show the full moon image (taken in my backyard BTW.) The third place grouping was Paris, which I assumed Paris France, a place I have been too but have no digital photographs of, especially on-line. The two images selected are definitely not Paris, France. So I tried investigating how to remove these photos from the grouping. The Google Photo Help only gave me a message it can't be done on a computer yet!  So I will have to wait, but not being able to help improve the classification

My last issue, and one that I haven’t seen this discussed elsewhere is how to find those things that are not being automatically found by google photo. If you take a lot of photo and they are being uploaded indiscriminately but the vacuum like google photos backup & sync utility, you photos could quickly become a mess, and it will difficult to find very special photos except by remembering the date a photo was taken. Again it would be very nice to be able to “seed” a new grouping by creating an album of the characteristic photos and then let google photo find other similar photos.

Automatic Grouping of your photos in flickr

In my recent post about the updates to flickr I put a teaser in about finding the Magic View (which you can find in the middle of the option header line under the camera roll). This is the most obvious application (it shows up elsewhere) of flickr’s new vision search engine, that scans incoming photos, recognises, colour and patterns, such that it can “understand” what objects and themes are shown in each photos. I am impressed at how well it performs, with only a very few misclassifications. If you already have a big collection of photos on flickr, magic view is the place to tame it.

The magic doesn’t stop there, even as you select select an image to upload (and the one on the right was uploaded from my phone the image is scanned and extra relevant tags are added (shown as unshaded boxes around uppercase Text) You can delete these is they are not approariate and at the sand time add your own

I have also updated the new flickr loadr but it still has the in discriminant vacuum cleaner style approach and thus there is less control of tagging and you just need to accepts the fact that the photo will will probably only the tagged with the most common and specifically most popular subject tags. However once uploaded it is still easy to group, retag and assign albums in the conventional organizr

The final place you will notice this advanced image analysis at work is in the search function, It will detect suitable images that match the theme even when the photo is not tagged with that theme. Again it works very well especially of standard popular photographic subjects like sunsets (I didn’t realize I had posted so many)

There are also some “advance” search options like colour matching, orientation and size.