Three Weeks On …the cloud photo archive splits into two

              

My cloud photo archiving experiment is continuing but only one of the original contenders, OneDrive, has survived and still running well. I was looking at a way to automated the update process to it and I have gone back to a very “manu-matic” (part manual, part automatic) method utilizing the powers of PhotoMechanic. There is a story behind that choice so I might explain that now.

A real advantage of the OneDrive approach is being able to control the sharing of specific photos with others in a secure (email private perma-link) connection (Dropbox also does this well). Google on the other hand isn’t secured (their idea of secure is just to have complicated file names, but that file name does have a permanent url, which is always publically accessible) AND I’m sure their approach will soon be, if not already being hacked. This private share feature would make such a service a good place for a family photo resource, that is only shared with the “trusted” family members.

This then got me thinking about making the family photos archive separately, and the more I think of it the better this strategy becomes. The difficulty is how to separate out the photos and more importantly the family just wants one or a few good photos not everything. The solution is good rating and culling. Damn, more work! This can be done with lightroom, On1, Aftershot Pro, Darktable using star & colour tab ratings. Providing I have this software automatically writes out. xmp sidecar files, I can make and change these rating at any time. These programs all monitor the .xmp rankings (Pick flag are an exception which none of the software seems to want to exchange, and I don’t use them anyway)  As I am reviewing or processing my photos I consciously mark those photos with the purple colour tab (pressing <Alt>5 in most software, but not lightroom). However I like to recheck my ratings and colours using PhotoMechanic’s contact sheet display view perhaps once a week, I review each days contact sheet (PhotoMechanics Term for the folder/thumbnail view) and I can then filter to see only one star and above rated. I can also check the colour tags and metadata updates if required. and recheck the ratings (eg change * to ** etc.). Next I remove the purple tagged family photos (they will go into a separate archive (see below) and finally I “save as” the selected group to my OneDrive folder on my computer. I am only saving finished jpeg files and have adopted the practice of resizing the photos to 2400 pixels on the longest dimension and jpeg quality to high. This approach to reduces the file size but in future being able to print approx 10” by 8”.  Finally it wpould be possible still be able to upscale because I have not utilized the .jpeg lossy compression to reduce size.

There was a secondary benefit from reaching back to a manu-matic step, after uploading my better photos into the cloud archive, without family photos included. I could switch the filtering to only include family photos (the purple colour tag and 2 starts of greater) and upload those to my trials of a private flickr family group and revised dropbox family folder. More is likely to unfold with this second type of cloud archive, so say tuned for updated.

Archive Obsession Three :: Building a Cloud Archive

There are a multiplicity of clouds services offered at the moment. It’s a bit scary because it would take ages to properly access them now. So this is a relatively incomplete list, but it is enough to get me started.

• Free “generic” Cloud Service
• Drop Box
• Google Drive
• One Drive
• Photo Focused Services
• Google Photo (its free and uses Google Drive)
• Flickr (it can be free)
• Smug Mug (starts at $3/month)
• Adobe/Creative Cloud (starts at $/10)
• DYI Cloud Backup/Archiving
• Back Blaze
• Carbonite
• Amazon S3, Glacier

It’s a no brainer to start with a test of the concept on the free services, especial as I already have experience work with all three. Not as a backup oriented services but I have used Drop Box as a way to automatically off load photos from my phone and also exchange big data sets with co-workers, when email struggles to handle the large files. Google Drive I use for blogger photos and as a cloud place to store training material and key documents while I was travelling. One Drive has a bit of a checked history (mainly because Microsoft was changing things a lot for a while there), the service looks nice and stable now.

The last two tiers I will probably leave to evaluate till much later down the track. I need to get comfy with the services I know how to use.

                                   

So beginning today I’m building cloud archive on all three Dropbox. Google Drive & OneDrive. I’ll let you know in a month or so which, if any suits by archiving requirements

Archive Obsession Two :: What makes a good Photo Archive

I know I bang on a lot about the difference between an Archive and a Backup. The key difference is a backup can consist of anything, even a great messy mass of files, whereas an archive is a clean long term place to store and easily retrieve information.

rchives, to be useful must include straightforward ways to find and view photos relatively quickly and easily. It needs a logical structure and whilst an archive should seldom be required, it should not be difficult to use or complex to learn. In other worlds KISS (Keep It Simple). It is desirable that it does not contain duplicates and damaged or corrupted photos. The essence is the collection, whilst it can contain a wide variety of photos and style needs to be clean and consistent. Archive should be managed and will grow with time. Backups should be kept of Archives

ackups, their purpose is to quickly restore the mass of data effected in the event that the media on which it is stored is damaged or lost. The emphasis is usually on a full restore, but a few systems do allow for a selective restore. Backup software seldom concerns the structure and viability of the data/files included. A corrupted file will be backedup the same as a good file. They are also focussed on the short term recovery of data. Old backups can become a liability if their contents is not known &/or well organized. Worse still they might be on a media format no longer supported.

Most photo oriented online discussions at present revolve around keeping backups of photo collections not archives. Certainly backing up your working data is important and the 3-2-1 approach is a good starting point. However if your collection of photos is getting large and you want them to be preserved in such a way that you can easily find a specific photo now is the time to start think about building an archive

Features I consider important in an Archive

1. Photos need to be indexed and catalogued (via metadata)
2. Photos should be easily viewed
3. Collection should be free of duplicates
4. Collection should contain only decent images (out of focus, corrupted or low quality images removed)
5. Photos should be in a limited number of standard (non propriety) formats

This is a pretty straightforward set of objectives but alas I have struggled to find a single commercial system that has these features. Lightroom sounded like it might be suitable but I found it both wanting and frustrating to use when several computers and people are involved.

• I have given myself the added burden of quarantining the Archive away from the internet and even my own local area network and housing it within a Linux workspace.

At the moment I use a Hybrid collection of software tools, I now used mainly PhotoMehanic to ingest (upload) from camera cards (I used to mainly use Picasa) and straightway cull the poor quality images and add appropriate metadata. I usually have a short break (could be up to a day or so) and then come back to PhotoMechanic and rate and colour code the images. I use a simple 1, 2 or 3 star rating, photos with 2 or 3 stars are the ones I would like to further process. They will normally be the ones I take into Lightroom (Aurora HDR or On1 RAW these days). Each step in Photo Mechanic is set up to record in both the Jpeg EXIF data and the .xmp side car files for RAW formats. The one star and above photos are the ones that get archived, any unrated photos will be left behind. At present I store the native RAW formats (.cr2,.pef,.orf & .dng) exactly as uploaded from the camera but I’m not convinced this a great long term option. Whilst there is some form of propriety CRC checks in the .xmp files I also create a file in each folder being archived with the .MD5 hashtag for each file. These CRC style checksum are useful to find duplicates and also to check with the files has been in any way altered (ie corrupted)

At the Linux end I read the same files and folders on external hard drives but now I have Linux version of XnViewMP and Aftershot Pro (both of which can view RAW files and read the metadata and ranking in EXIF data and/or the sidecar files. Both these programs can build catalogues or just browse folders. I have not yet tried to catalogue the whole collection but I’m slowly working towards that objective.

Archive Obsession One

My brush with a significant portion of my photo collection “disappearing” has left me with the uneasy feeling that I also need to make sure I have an Archive on a different media (ie not just spinning hard drives). At the moment I am using old hard drives taken from computers I no longer use (this is nice and cheap) but with only two cycles, since starting the air gapped archive approach, I’ve run out of capacity for my full photo archive. More significantly they are old and mechanically, thus liable to eventually fail. The fact that this archive can be accessed from a machine that is not on the internet means that they are largely protected from malware, ransomware, like wanna cry, and viruses. Even though they aren’t spinning all the time, this does not rule out mechanical failure of the drive but it does reduce the risks at lot. As I am only updating the hard disk archives every six months, mainly due to my self-imposed rigorous archiving regime, I only have two cycle a year on which is probably not often enough.

About 1/3^rd of my collection is already archived (in two places) on CDs and DVDs. This cover the period 1999 to 2010 and for most of the period to 2013 after that, but I have only one set of DVDs offsite. Thus on the old hard drives I only have to cover my archive from 2011 to present (just under 2TB, terabytes). But due to the lesser capacities of my re-purposed of storage drives, I have had to split the archive across two drives (eg 2011 to 2015, then 2016 to 2018), So everything is covered but it is still a bit messy.

Thus it is time to consider an alternative media (and/or methodology) to suit my growing photo collect and the growing chorus is suggesting to look at the cloud. So over the next few post I will be investigating -

1. What I expect an archive to be.
2. Suitable cloud services for Photographic Archives.
3. The “hidden” cost of archiving.

Thus begins a major Backup & Archive Overhaul 1. Archiving

The Suspended animation Windows10 Death By upgrade of two of my three current computers, plus another taken out of retirement really frustrates me but it has also forced me to carefully rethink my Archive and backup systems

First Archiving

I have often made claims in this blog that Backup and Archiving is different but haven’t really explained why and what I do?
I use a really old fashion 3 generation approach, Grand Father > Father > Son which originally came from using large real Magnetic Tape as the archive of mainframe data on larger installations.
The system is simple, once a month (or at some routine time, fortnightly, bi-monthly, quarterly, yearly) a new master archive tape (or Tapes) is written and it sticky labelled SON (on the outside not magnetically). The next month a new master is written on a fresh tape. The SON Label is removed from the old tape and affixed to the new master (it could be a new label). The older Master is now labelled FATHER. The Next month a new tape  is written and label SON the old son becomes FATHER and the old father becomes GRANDFATHER. It was common to keep either the Father or Grandfather versions offsite. On the forth and subsequent months of the cycle the GRANDFATHER tape is overwritten with a fresh new full archive and becomes the SON. This approach means that you can recovered items up to 3 months old even if the where corrupted (or lost) on recent archive. It also Has the Advantage that the tapes are being regularly exercise and not slowly magnetically leaking data/corruption across the layers of the tape.
For the past few years I have followed this approach but instead of tapes I’m using old hard disk drives recovered from superseded computers. I have a simple USB plug in SATA HDD adaptor (they are not expensive) and the old mechanical hard disk are essentially free and reformatting then re-using them stops your data falling into the hands of unscrupulous computer recycles and being sold to hackers or phishing schemers. Two years ago now in my first set of frustrations with Windows 10 update I gave up trying to install windows 10 of a small Toshiba netbook (and instead installed LINUX on it and it works fine). It is now forms the key component of my Air Gapped Archive, its not top secret spy style setup, just a way to try and keep viruses and normal mechanical malfunction that might affect my normal computer network away from my archive files (and I’m well away from any internet connections).
I have dropped back to doing the full archive update quarterly but I do maintain a candidates for latest archive area on one of the disks on my local area network that all computer can copy to. Each quarter I transfer this to my Linux computer via an external USB drive and sneaker net (ie I unplug it from one and plug it into the other). This external drive is itself a backup of the latest SON Archive (perhaps recreate then and there) and so begins the process of cleaning and deduplication etc). This can take a from a few minutes to many hours if not days for poorly organized collections of everything!
Just have everything fully backed up does not constitute and archive. An Archive needs to be organised so things are easily found (keywords are not enough). This is a big topic and will be the subject of a few post to come.
Once I am happy with the new additions to the archive I merge then into the full son set of directories, I then get the grandfather drive which is normally keep offsite so I need to collect it or tale my Linux laptop to the offsite location. I then reformat the grandfather so it is basically a fresh clean disk. Exercising (aka user) a mechanical hard drive actually improves their life span and significantly reduces the likelihood of data loss. This reformatted and empty now gets a full copy of the working son archive labelled SON. the old son become FATHER and old father gets labelled GRANDFATHER and retired offsite.
This might sound complex but it is easy, and extra reliable. Despite being 2 out of 3 computers down I know my archive is safe.

Memories

On the flight across the ditch I watched the film called " A Month of Sundays", its a quirky poignant story, yet honestly Australian. One of the important subtexts is about the significance of memories. I'll say no more not to spoil your viewing. It has me thinking about what are memories and how we might make them? I live in a very bush fire prone region and it is an unwritten law that if you need to evacuate the first thing to pack is the photo album. The theory being that they hold all your memories.

However photo are just images in tone and sometimes colour. Some say nothing but the special few will evoke a fondness in the viewer and bring up clearer pictures of the place and/or the people, the smell of the garden or the sensation, of intense heat, or being drench by driving rain, the special treat, ice cream or toasting marshmallows. The memories spread into many more dimensions than the flat photograph. These are indeed precious things. However how do you create them?

I'm starting to think you can't. They usually create themselves.

As photographers we are constantly being told to always tell a story. Which is a real challenge in a single still image because stories have a beginning and an end and time between that moves step by step between these end points. Some still images might imply the beginning and how the story might have arrive at the end depicted in the photo or alternatively that might capture the start and the end is implied. A very few photo capture one of the steps.

To tell a story properly you need words and or a video. The reading of the words of watching of the video compel the viewer to concentrate. The mind is generally supressed from wandering into memory until the story is over and your concentration is released.

What I see happening these days with so many of our photos and snippets of videos online, is not about memory. These are instant distraction, kind of a visual gossip of things (in the real world) that can be scrolled through and laughed at. They are short term relevance and mostly soon forgotten

However older photos particular acquire a magical property to bring up deep memories in the viewer(s). So when and how does this happen? I think there are two important components. They are mostly physical (a print or a film strip) as opposed to temporary light on a screen) they have survived the passage of time, not something thrown away or discarded and forgotten. Thus they somewhat rare and precious.

I really doubt any amount of cloud storage and interconnection will achieve this. That all knowing science fiction screen on the wall will certainly have lots of data and images but it will not have obvious precious things. They might be there but they will probably be hiding.

So who will save us? You might be surprised, we might already have a new breed of memory managers amongst us, usually they are younger women, perhaps recent mother who are both social media savvy and at the same time deeply embedded in the family structure. They are the ones that are curating the "look at me" stream of friends and family on social media and making an effort to comment on and save the important parts of the puzzle that will house our future memories. The problem for now is there are few tools to make this easy and these new bread of memory managers, because at the moment they are unlikely to have a deep knowledge of file formats, the archiving strategies or digital asset management at all. If its not a physical print (most just collect 6 by 4 prints) or jpeg files (unfortunately those heavily compressed optimized for web efforts). They will probably have any metadata stripped out by whichever social media service they got momentary fame on. Yet I hope these memory managers survive and flourish. Find each other and build better strategies and places to house our memories. So that in fifty years there will still be something to save, hopefully in a format we can still read, from the approaching wild fire and it will be precious.

Prints can be Excellent Archives as well

I have come to accumulate a lot of very early family photographs and they are still in reasonable condition (some aren’t more on that in another post). The biggest challenge is figuring out who is who (picasa face recognition is helping but it does a fair bit of grouping parents with their children and mixing siblings etc) and getting things organized, and working out safe storage. Not such hard task but time is required. However I have begun to imagine what a horrendous task this might be for future generations that might suddenly find themselves with a few million digital photos many in camera specific RAW format, spread across a number of the by then outdated USB external drives or in a massive zipped conglomerations plucked from a social media web service just before it shut down. The solution to ensuring a future of family photo legacy is simple, just make prints of your favourite photos. Print those important family photos now!

This brings me to how to get those prints of all your social media post (the ones you really want) onto prints (or photobooks). A couple of years ago that could have been expensive but several services are gearing up to make that more affordable. Services like Big W, Harvey Norman and Ted’s camera have kiosks that will let to log into a number of popular social sites, like facebook and instagram. On-line service like shutterfly and snapfish (there are plenty of others) offer some compatibility to pull out images from your social photostreams. However often these will probably have been resized to suit fast web display and might not be able to be printed much larger than 10 by 15 cm (4 by 6”).

Whilst I do have my own ink jet printers (yes I have a few including a decent older HP photosmart with 12 ink colours but now I’m having difficulty getting the ink cartridges) I usually get prints made from an on-line service such as snapfish. The reality is they are cheaper by the time you consider the ink and paper costs (and less frustration) up to about the A4 size. Current prices of the better deals should be around 15 cents for a 10 by 15cm up to just over $5 for an A4. Many services still offer the 10” by 8” inch size as the largest mass processing option at around the $5 mark. These are the size ranges that will suit most family legacy collections anyway.

I notice that snapfish are heavily advertising unlimited 6 cent prints at the moment on TV and this is a great deal to start you family legacy project. If you miss this deal just keep your eye out for other similar offers, many services offer 50 to 100 prints free when you sign up. This lead me to look at printing from flickr (which I have done previously to snapfish on many occasions but the links from your photostream now go to the more expensive wall art and photo book services offered by flirkr itself. However creating prints is still there in the organizr tool under the print and create tab. You need to scroll past the wall art and photo book option and then you will find other options, including prints. In Australia the only non flickr photo services come from snapfish.

Digital Archives :: An Unfortunate Truth

The biggest issue in setting up a long term digital archive is related to what physical storage media you plan to use. The digital world is littered with various storage media that have very short operation lives and/or have fallen from common usage. I have used Punch paper tape, Punch Card, Reel to Reel Magnetic tape, Various Tape Cartridges, 8”Floppy Disk (yes they were once that big but they could hold around 1.3mb), 5¼” Floppy Disk (only 360kb), 3 ½” Rigid disk (720kb but later 1.4mb), Zip Disks (a glorious 100mb), CDs (compact disks with a capacity of around 700MB) on to DVD (which generally hold 4.7gb) and Blu-ray (which can hold up to 25gb). Unfortunately the unspoken secret here is as the capacity and specifically the information density has increased the shelf life tends to decrease. The punch paper tape for the late 60s is good to go (if only I had a reader) if it hasn’t been torn, so are the punch cards from the early 70s, but everything else has probably reached its best by date and there is a reasonable chance you will experience corruption or errors reading older media already. Even CD and DVDs, once thought invincible start to have troubles after only a few years, even less if they are not handled carefully. What about solid state memory (like SD memory cards, USB keys or the newer SSD drives), unfortunately they have limited life spans as well (more to do with the number of reads and writes) in normal usage they may outlast the next form of storage HDD. The conventional hard drives (whether built into your computer or as an external USB style) also have some telling untold secrets (see backblaze study, which suggests seagate drives are more likely to fail than western digital, but I have had the opposite experience I have 4 dead western digital my book USB drives but 3 healthy expansion seagate drives, one of which considerably older). The caveat here is they will probably last a day or two longer than the guarantee. So what about cloud storage that’s forever isn’t it? Well if you look in the terms of Service (TOS) or most cloud service providers there is generally no mention of loss or damage of stored data (I guess they assume you must already have a copy as well, ie from their point of view the data is backed up by you). Only Amazon seems to have addressed this, and I hope companies like google could be relied upon. However they are companies that are unlikely to last forever. So cloud backup is where we are at now, not cloud archiving yet. So where does that leave us. We will need to plan to regularly move any digital archive onto new media as technology change and popular taste determines. Clearly an Archive of Digital Photos must not be hardware/media specific.

I trust all this scares you into making a backup of your digital photos right now. Stop reading and go and do it!

Archiving Long Term, a timely warning

Vint Cinf recognised as one of the fathers of the internet, recently warned of a coming “digital dark age”, which was picked up by the UK Amateur Photographer magazine in a nice article titled “Photographers are sleepwalking into a ‘photographic Armageddon’“ I must admit I fully agree with the issues raised, the media we rely on today to store digital images does have a limited life, probably shorter than we realise.

My first digital images where recorded using a Sony Marvica Camera on 3 ½” diskettes and whilst I have a few of the diskettes still I think I might be struggling to find a diskette reader, and that has only taken 15 years. I did have the foresight to back them up, but my preferred media then was a zip drive, Damn yet another obsolete format! Ok I have taken some photos into my main photo archive and most have already been recycled onto CD (but already computers don’t always come with a CD/DVD drive, phones & tablet certainly don’t)

The essence of Vint Cinf talks was that photographers also need to print there work to ensure its long term survival. However even with the best archival papers or other media the longevity is never guaranteed yet it is certainly a better choice for important family snaps. By Prints, I’m sure this doesn’t necessarily mean the same old 10 by 15cm snapshot (that’s 4”by 6” for the metric-impaired) you can store in an old shoebox. in could be printing to a photobook. If you want to do you own printing, remembering many ink jet printers use dyes rather than pigments and the colour will quickly fade if exposed to daylight. (ie display on the fridge door)

The morale of this story is make sure you include some form of hardcopy print to provide an additional archive security for your most cherished photos (better still print two copies or have one print and include the same photo in a photo book). If you intend to fames and display the print, put it behind glass and/or have a second print stored safely.

Taming duplicates

I have had the exclude duplicates item ticked on my picasa imports by default, for the longest time. This helps avoid a lot of duplicates turning up because some photos where left on a card or a phonephoto was uploaded via wifi more than once or via different services, However because I now have my photos flow across several computers and software packages it is surprising just how many duplicates do turn up. So I have also been using a neat little utility program from digital volcano called duplicate cleaner, which is design for all file types but does work very well with media files like photos, video and music. The biggest feature of this cleaner is that it can recognise the same file contents, even if the name is changed. If the file is a jpeg and has been post processed it will be considered as different. Also if the jpeg has been rescaled it will also be considered different. This is fine I probably want to keep these,

During and particularly as I reach the end of the month I move my photos from the collection points (laptop computers) onto a backup drive and it is at this point that it is most useful to run the duplicate cleaner (if you have a lot of files it can take a while …so go get a coffee). After this I then copy the backed up files to the archive drive and make my DVD/CD backup set of the archive (this is my one remaining traditional backup, everything else is now on portable hard drives). I don’t delete the photos on the collection computers until I have finished the backup –> duplicate clean –> Archive cycle. The very end of the month is the right time to do this.

Letting My Mistresses into the Office

With the resent death of a couple of my hard drives located in my “comms room” (actually a cupboard) and/or the studio I had to rethink what was attached to my LAN and where it was located. This meant simplifying things reducing the number of network driver and bringing the router, ADSL internet modem and network drives into the office and removing a lot of cables, a patch panel and even the Fax! Leaner, meaner and easier to manage now. I still have a RAID system (a Netgear Stora) to keep a mirror of all the working directories around my Network, but I don’t store music or photos on it anymore. My Photo Archive now sits on an external hard drive which hangs off the main office desktop (it is the one computer I leave turned on most of the time). Since this is where the archive lives I figure I needed to have an index or catalogue (not in the lightroom sense) of what it contains also on the office desktop, but that is the catch there is not one system that can really managed my them all.

So….

If you look on the upper right hand side, of what is otherwise a very business like place, you will see a few of my photographic mistress have moved in. Its started with picasa, which is on all my computers, but soon included lightroom (to better handle the RAW) and then XnView (mainly to bulk add/update EXIF & IPTC metadata), AfterShot Pro (faster than lightroom) and then Perfect Photo Suite (its 64 bit and doesn’t work on the studio computers), So I have two teams now, the jpegs and quick look stuff (Picasa & Aftershot Pro) versus the RAW and more intense pair (Lightroom and Perfect Photo). I had to turn off RAW viewing in Picasa, to force it( and me) to focus on JPEG but things seem to be going ok (for now!). I tend to avoid too much post or re-processing here, instead I either export jpegs or make a sub-collection of RAW file and export them as a lightroom catalogue backup, to be used on my laptop or studio computers. Then reimporting the updated catalogue. This round trip to other hardware is not such a pain because the files are shared across a couple of share network drives (P: for the Photo Library itself & S: for Scrapbook, working projects etc).
As long as they behave themselves I figure these photographic mistress may stay.

More refinement to my A-B-C Photo Flow

Its the end and one month start of the next, time for the monthly backup. My A-B-C process, shown in a more physical photo flow back in April last year, is bedding down nicely. The A for Acquire is everything happening on the left hand side of by flow chart below. The B for Backup are the arrows to the items on the right. The C for Collection (aka arChive) now sit in a network storage device (the red NAS drive on the top right). I have also added in how I use Dropbox and Google+ Autobackup, which I intend to discuss in separate future posts
I have added in the devices and software I use at each step. I have three laptops, which serve different tasks but all act as photo collectors, each of these store the photos and videos together organized by date in sub folders. I often will make incremental backups during the month the one of my portable USB drives (I have two and use them in rotation), particularly when I am travelling or when I have taken a large series. Only one desktop computer is now involved in the archive component of my photo flow and it is used to hold the catalogues of the photos (and videos) in my archives on the network drive, so I update these catalogues as I backup to it. Whilst all computers on the network can access those archive libraries (as networked folders) I have now opted to have three separate catalogues (in Corel Video Studio it is more accurately called a library) because different software offers different advantages for specific formats. How these catalogues get back up is also the potential topic for a future post.

Keywording your photos

eywording (or tagging) your photos, sounds simple enough, but if you haven’t already started the task could have already got away from you.For someone taking and preparing a lot of stock photographs, some form keywording is probably essential but the time required and benefits trade-off for the rest of us will not be so clear. I take a wide variety of photos anyway, many are just for personal use, for painting reference, for use in collage or panoramic and a lot are family etc. My most important task is to to make these separations in my collection, I could do that physically by storing them in separate folders or even different hard disks. However, using key words and simple filters with do the same job from a unified collection (see my strategy 1 below). Designing a perfect classification or keywording system is a really broad topic but a good start (I’ve been using this approach for a while, but I have to admit not everything is keyworded yet) is to use a hierarchy based on the classic Who, What, Where, When, Why and How.  You probably won’t uses these categories as keywords themselves but they should help guide you as the most likely way to “find” an individual photo in the future. The Who? is pretty straight forward, and there are a number of ways to tackle this but a person’s name is usually fine, but this can be a lot or work for big groups/events, Lightroom affectionardos might like Gerard Murph’s method of custom keyword sets. The What? is very obvious you should describe what is the subject of yoour photo. This is likely to be the biggest part of your keyword list and think of this as a hierarchy and including each term for each level of the hierachy (see my strategy2). I remember hearing Varina Patel discussing this on a podcast and giving good advice on keywording by theme particular if you intend to submit photos to stock agencies. Several of the keywords like the When? and Where? will be best to be automatically read from the Cameras EXIF files, or batch such as GPS coordinates in a merge of .gpx files. There is also a lot of How? details also available in the EXIF data, but a few special keywords to relate to technique won’t go astray. It is the Why? that most people probably overlook, but the reason you took the photo in the first place is likely to be the quickest way to find an image!

There is a massive trade of here, adding keywords will undoubtedly make photos easier to find, but how much easier and how munch time is saved is pretty much an unknown at the time you are supposed to be doing the keywording. Against that is the amount of work you have to put in tagging and classifying your photos.In particular everyone will tell you you need to do all this extra work up front. Otherwise it becomes too large a task to tackle. So how do you get started, here are some good strategies I have found

1. Have a few (less than 12) broad categories (eg. Portrait, Landscape, Family, Event) so one of which at least is used for every photo, and applied as or soon after you upload your photo. These should be based on the themes and types of Photograph you like to take. If one category gets particularly top heavy consider splitting it into some sub-categories and always add those as a second keyword.
2. Aim to use at least 3 keywords for every photo that correspond to different levels in your classification Hierarchy (Its simple maths the more levels of categories you use the easier it is to find a specific photo). Yet be constrained 7 or more keywords may not give you much advantage in the end.
3. When creating new keywords, avoid plurals, compound hyphenated terms (it is better just to use the two words as individual tags) and ambiguous terms. Watch out for misspelling.
4. Use albums (or Collections in Lightroom) to subset important groups when attempting to batch recode keywords into exiting photo collections. The big advantage is you can “see” the collection together. Taking time to make special collections and batch keywording them at the same time is the quickest way to keyword unorganised photo archives and avoiding this being a chore.
5. Tagging your photos with your keywords, as well as embedding them in your EXIF metedata when you post them on social media for public view if a great way to get extra traction  Unfortunately once again there is no uniform of standard way to do this, but it is worth investigating your favourite publishing places and services (eg in lightroom publishing services you can add metadata and customised presets).

Rethinking the Longer Term

rganising you’re your photos is very much about which software you use to manage them, but everpix got me thinking about how you can access a large collection possibly when the current crop of software is out of date, no longer supported or just not compatible with a new operating system. It is very hard to second guess which package gives you a good archive life. The sad reality is you may do better with printed hardcopy than an amorphous mass of digital files (possibly of different formats). There are very real example of the difficulties to come. Do you still have players for your vinyl records, cassette or 9 track tapes, what about the Beta Max or VCR video tape? Can you read that 5 1/2" floppy disc of ZIP cartridge? We are junking our "less visible" storage technologies at a great rate. DVDs, Thumb Drives and External USB drives will probably go the same way.
So How might we best future proof our collection?
My first thoughts are

1. Make your collection more easily searchable
2. Stick to the popular formats
3. Store the originals unaltered (and remove duplicates).
4. Archive only the best of the post processed photos.
5. Understand the need to refresh (and move) your archives

Over the next few weeks I hope to write a few more post on this and related topics

Thanks to Jessica Hische for her great drop caps.

Yet another iTunes related death

I really have had a bad run with iTunes. After resisting months constant prompting to update, I gave in yesterday and updated the one version of iTunes (version 11) that I still run on my older studio laptop (still running Vista). I only have it to manage my old iPod nano, and specifically so it can manage podcats. For a long while I could not find any other easy way to do that. Now mainly because of previous disc crashes I don't let iTunes near my music collection any more and I have discovered stitcher radio.

True to form I had just downloaded and installed the update (iTunes version 12), which involved restarting the computer. It restarted once but crashed within a minute and this repeated about 3 times then I got the warning that my system was unable to start (well that's stating the obvious). After about 2 hours of repairing this morning and restarting I did get my system back, albeit without the update to iTunes. So the first thing I did was delete any trace of iTunes.

I wonder how my iPod will go as an orphan, and I'm again reminded why it is so important to keep good archives

Hard Lessons on the Importance of Backups & Archiving.

In addition to the picasa database mix ups, which may or may not be related, I have suffered three external hard drive crashes, or malfunctions, in the past year, All of them contained photos but fortunately I didn't loose anything! I have been very regular with creating backups and maintaining at least 2 archives.

As all three of the drives were also holding my music library which iTunes was using at the time of the crashes AND there are so many hits on the net offering advice about recovering your iTunes library when you disc crashes (more often than not in iTunes 10). I started to get suspicious about iTunes 10 being actually responsible. No music was lost but I now rely on windows media play to listen to my music and just use iTunes 11 (which I really don't like) on an old machine to get podcasts and manage my old iPod Nano (which I still like)

Saving face, after a picasa database corruption

I spent a lot of last weekend carefully timing how long it took various programs to load my photos from cards. Picasa was the outstanding performer by now it has let me down badly. It refuses to start on my main photographic computer at all just giving me this error message and if I say yes, reporting an error with that specific file.

I have picasa on several other computers and two of them, both updated last week are showing occasional signs of corrupted data base problems (ie they often display the wrong thumbnails for recently uploaded photos. This has a frequently report problem since 3.8 but something has clearly made this worse on my computers. I suspect the 3.9 update. The recommended way to fix a corrupted database in picasa is to firstly delete picasa and then reinstalled the software a fresh. Yes that’s what it recommends!

This is fine for most features because the editing processes for your photos are actually stored in the picasa.ini files that are located in the same directory as the photos. the central databases just storing a thumbnail of the photo and its location, EXCEPT for some details if you are using face detection feature and any albums you have used. The flag that a photo might contain a face is stored in picasa.ini file BUT the detail of who is who (ie the names you have given the face are actually stored in . I suggest you might want to read the well prepared tutorial "Saving Faces When you Re-Install Picasa"

As I am in the middle of rethinking how and where I storing my working copies and archives I figure now is a good time to bit the bullet and re-installed picasa a fresh and get it to re-index only the photos and directories I want. I guess if it all goods well you probably won’t hear more in here BUT I am a little worried its going to be more complex than that! Things didn’t great off to a great start, when I tried to report why I was uninstalling picasa, I repeated got the following error!

See also my Un-Jumbling picasa update

ABC of my photo flow

I have been quietly reviewing several aspect of my digital photography, including the physical aspects my photo management, what I do and what I have to carry as I travel around (a couple of trips are coming up). So I thought I might share some of this here.

A for Aquire

This is the step where I first take the photo and then transfer it to some other storage device (typically a laptop or net book computer) . I have two options here either by removing the card from the camera, or connecting a cable. Wireless or Net based transfer could be used here as well but I'm lagging behind the times a little in that regard.
 

B for Backup

This is the step I think a lot of photographers over look but it may be the most important. At the same time (or very soon after) you transfer your images from the card or camera you should also make a backup copy (doing this automatically is best). At my office I have a Netgear Stora (a duplicate drive raid system) that is set to automatically backup everything from the local user drives on my network. So loading photos into my picture folder, for example, automatically creates two copies on the stora. When travelling I usually have a small USB backpack style hard drive and I manually copy any photos uploaded to it. Whilst I have gone from a single copy on my camera/SD cards to two copes on harddrives, travelling is a higher risk time for this style of storage, so just in case I carry a couple of empty USB keys in my camera bag to make sure I have additional backups of any "keepers". These backups copies are short term and can be cleaned out once the photo sets go into archive collections.

 

C for Collection

My Primary Copy/Collection, the long term access and storage location, the true archives, are updated regularly usually monthly, but sometimes after special trips or events. I have an permanently running external hard drive connected to my LAN and which contains a master PhotoARCH folder, this is shared between my all my computers, and under that folder are subfolders for each year, then subfolders for each month and finally sub folder for each day.  I have found archiving by date is the easiest way to organized my new digital photos. At the same time I update the archive I also make external copies to optical media (eg DVDs). In the past I made two copies of the DVDs and stored one off site. Recently I did some sums and it is now actually cheaper (a set of 50 DVDs for $24 means the long term cost of storage is around $0.11/MB whereas a 3TB drive for $120 means storage is $0.04/MB) to use a large external hard drive just loaded with your photos and sitting on the shelf, compared with the same storage sitting on DVDs taking up the whole shelf (3TB is equal to approximately 670 DVDs). So now I have two external drives for my archives, one on-line (just on my network) all the time and the other on my shelf. I still make a DVD set to store off-site each month.
If you are a fan of the 3-2-1 approach to backup you will notice that this system incorporates those principles (3 copies, two media and some physical separation between the copies.) I prefer to call it my A-B-C.

A slow death (or lessons in archving)

I have been using a 1TB My Book USB (backpack) hard drive as my principle archive location for my photo collection for several years now and it has worked very well. It is connect to PC being used as a media center so it is always on and has a wide screen LCD TV to view the photo directly on that computer (from the comfort of a couch, not quiet but close to the old 35mm slide show feel). The TV screen is a HD flat screen Sony Bravia, connected via a HDMI cable, so the photos are crisp, clean and at a good resolution, I had observed that that drive was getting slower and slower since I also put my itunes library on that disK. Terminally slow! Then quite by accident I noticed that I could not read a couple of photos, then another and others where so slow to respond. I decided it was time to copy everything to a newer backpack drive. Half a terra byte of photos and videos takes a fair while to copy (off a drive that is slowing anyway) and in that process I discovered a dozen files that I could not read. It was easy to find good copy in the two sets of CD/DVDs I make each month as backup. So no data was lost but it has forced me to rethink the strategy of only one archive on-line (ie on a constantly spinning hard disk).
A single library of photos is much easier to maintain. I’ll have to report soon on another experiment I had tried where I used different software on different computers to test their management capabilities. While the programs all worked well using a range of system was a confusing disaster, So I like the idea of a single “central” archive BUT that means the risk of only having a single copy. Using a raid system with mirror backup is one good option, and I do have a netgear stora for my business and thus has worked perfectly however it doesn’t really have the room. Another option is to use an online back service (i’m trying out iBackup, but there are many others). The third option is to use have two backpack drives and have your own scheduled SYNC process that routinely copies files to a second location ( I'm going to start with the reformatted my book on my home network for this.)
So which option have I choosen?
Despite the hard lesson learnt about using multiple photo management packages being a disaster. I will be using, or at least trying out, all three options, I’m splitting my collection into three 1. family & events (using a daily backup schedule to a secured private net location) privavy is important here.  2. video, the most rapidly expanding aspect of my collection (synced to the My Book, because the media center is where I’m most likely to view and work on video) and 3. everything else (mirrored on my stora).
Finally I’ve decided that my archive is not an archive at all, its my original collection, the “backup” CD/DVDs are really an archive and the the second live copy is my true backup.

Home Sweet Home

As well as trying to keep down the amount of Photo and Art stuff I carried while travelling I also put myself on a software diet. I was travelling with a new but very small Toshiba notebook and I wanted to avoid the over installation of unused software glut, that inevitably slows down any computer (well any of my computers) eventually suffers from. So I deliberately only installed Picasa and some Pentax drivers (notably Remote III so I could tether my camera, which I never did) and a few utilities on a Portable App memory stick (I used Autostitch a lot , Picturenaut and  Noiseworks  less so). The one thing I didn’t have which I might have used (and tried unsuccessfully to achieve) was creating massive panoramas and photosynths. Admittedly you can do the photosynth thing by any browser but uploading all the images could take forever on a lower bandwidth connection. You also have to download some software. So I was glad to get home, get my photos onto my stora (there archive location) and do a bit of reorganizing with Windows Live Photo Gallery (again I could have downloaded any of the windows live stuff while I was travelling, if I got desperate). Finally I was glad to get back to a dedicated flickr uploader facility, rather than wait on the web browser based upload tab with flickr. What I enjoy most of all was the wider bandwidth

Notice I haven’t loaded either Photoshop elements of Lightroom onto my desktop yet.