Sunday, May 20, 2018

Archive Obsession Three :: Building a Cloud Archive

cloud, images iconThere are a multiplicity of clouds services offered at the moment. It’s a bit scary because it would take ages to properly access them now. So this is a relatively incomplete list, but it is enough to get me started.

  • Free “generic” Cloud Service
    • Drop Box
    • Google Drive
    • One Drive
  • Photo Focused Services
    • Flickr (it can be free)
    • Smug Mug (starts at $3/month)
    • Adobe/Creative Cloud (starts at $/10)
  • DYI Cloud Backup/Archiving
    • Back Blaze
    • Carbonite
    • Amazon S3, Glacia

It’s a no brainer to start with a test of the concept on the free services, especial as I already have experience work with all three. Not as a backup oriented services but I have used Drop Box as a way to automatically off load photos from my phone and also exchange bit data sets with co-workers, when email could handle the large files. Google Drive I use for blogger photos and as a cloud place to store training material and key documents while I was travelling. One Drive has a bit of a checked history (mainly because Microsoft was changing things), the service looks nice and stable now

The last two tiers I will probably leave to evaluate till much later down the track. I need to get comfy with the services I know how to use.

                    box, drop, media, online, social icon        drive, gdrive, google icon        circle, colored, gradient, media, one drive, social, social media icon

So beginning today I’m building cloud archive on all three Dropbox. Google Drive & OneDrive. I’ll let you know in a month or so which, if any suits by archiving requirements

Saturday, May 19, 2018

Archive Obsession Two :: What makes a good Photo Archive

I know I bang on a lot about the difference between an Archive and a Backup. The key difference is a backup can consist of anything, even a great messy of files, whereas an archive is a clean long term place to store and easily retrieve information. A

rchives, to be useful must include straightforward ways to find and view photos relatively quickly and easily. It needs a logical structure and whilst an archive should seldom be required, it should not be difficult to use or complex to learn. In other worlds KISS (Keep It Simple). It is desirable that it does not contain duplicates and damaged or corrupted photos. The essence is the collection, whilst it can contain a wide variety of photos and style needs to be clean and consistent. Backups should be kept of Archives


ackups, their purpose is to quickly restore the mass of data effected in the event that the media on which it is stored is damaged or lost. The emphasis is usually on a full restore, but a few systems do allow for a selective restore. Backup software seldom concerns the structure and viability of the data/files included. A corrupted file will be backup the same as a good file. They are also focussed on the short term recovery of data. Old backups can become a liability if their contents is not known &/or well organized. Worse still they might be on a media format no longer supported.

Most online discussions at present revolve around keeping backups of photo collections not archive. Certainly backing up your working data is important and the 3-2-1 approach is a good starting point. However if your collection of photos is getting large and you want them to be preserved in such a way that you can easily find a specific now is the time to start think about building an archive

Features I consider important in an Archive

  1. Photos need to be indexed and catalogued (via metadata)
  2. Photos should be easily viewed
  3. Collection should be free of duplicates
  4. Collection should contain only decent images (out of focus, corrupted or low quality images removed)
  5. Photos should be in a limited number of standard (non propriety formats)

This is a pretty straightforward set of objectives but alas I have struggled to find a single commercial system that has these features. Lightroom sounded like it might be suitable but I found it both wanting and frustrating to use when several computers and people are involved.

  • I have given myself the added burden of quarantining the Archive away from the internet and even my own local area network and housing it within a Linux workspace.

At the moment I use a Hybrid collection of software tools, I now used mainly PhotoMehanic to ingest (upload) from camera cards (I used to mainly use Picasa) and straightway cull the poor quality images and add appropriate metadata. I usually have a short break (could be up to a day or so) and then come back to PhotoMechanic and rate and colour code the images. I use a simple 1, 2 or 3 star rating, photos with 2 or 3 stars are the ones I would like to further process. They will normally be the ones I take into Lightroom (Aurora HDR or On1 RAW these days). Each step in Photo Mechanic is set up to record in both the Jpeg EXIF data and the .xmp side car files for RAW formats. The one star and above photos are the ones that get archived, any unrated photos will be left behind. At present I store the native RAW formats (.cr2,.pef,.orf & .dng) exactly as uploaded from the camera but I’m not convinced this a great long term option. Whilst there is some form of propriety CRC checks in the .xmp files I also create a file in each folder being archived with the .MD5 hashtag for each file. These CRC style checksum are useful to find duplicates and also to check with the files has been in any way altered (ie corrupted)

At the Linux end I read the same files and folders on external hard drives but now I have Linux version of XnViewMP and Aftershot Pro (both of which can view RAW files and read the metadata and ranking in EXIF data and/or the sidecar files. Both these programs can build catalogues or just browse folders. I have not yet tried to catalogue the whole collection but I’m slowly working towards that objective.

Friday, May 18, 2018

Archive Obsession One

My brush with a significant portion of my photo collection “disappearing” has left me with the uneasy feeling that I also need to make sure I have an Archive on a different media (ie not just spinning hard drives). At the moment I am using old hard drives taken from computers I no longer use (this is nice and cheap) but with only two cycles, since starting the air gapped archive approach, I’ve run out of capacity for my full photo archive. More significantly they are old and mechanically, thus liable to eventually fail. The fact that this archive can be accessed from a machine that is not on the internet means that they are largely protected from malware, ransomware, like wanna cry, and viruses. Even though they aren’t spinning all the time, this does not rule out mechanical failure of the drive but it does reduce the risks at lot. As I am only updating the hard disk archives every six months, mainly due to my self-imposed rigorous archiving regime, I only have two cycle a year on which is probably not often enough.

About 1/3rd of my collection is already archived (in two places) on CDs and DVDs. This cover the period 1999 to 2010 and for most of the period to 2013 after that, but I have only one set of DVDs offsite. Thus on the old hard drives I only have to cover my archive from 2011 to present (just under 2TB, terabytes). But due to the lesser capacities of my re-purposed of storage drives, I have had to split the archive across two drives (eg 2011 to 2015, then 2016 to 2018), So everything is covered but it is still a bit messy.

What next?

Thus it is time to consider an alternative media (and/or methodology) to suit my growing photo collect and the growing chorus is suggesting to look at the cloud. So over the next few post I will be investigating -

  1. What I expect an archive to be.
  2. Suitable cloud services for Photographic Archives.
  3. The “hidden” cost of archiving.

Thursday, May 17, 2018

Saved by my Air Gapped Archive

I had a perfect storm of issues over the past week. I had the April Update of windows 10 forced on my computer and in the process the Homegroup networking just disappeared along with some of the network drive letter I used to share folders across my local area network. Also the NBN line to the internet (and my phones) dropped out and took many mobile phone calls to report and get re-established. NBN is not a popular word around here. When the internet was back the modem and router began playing up, WiFi dropping out and weird things happened across my network, including my Netgear Stora NAS (Network Attached Storage) which I use for backup/disk mirroring went silent and these was the pain of another few days of reporting problems. Long hours of automated phne system and support calls and just general frustration.  Then horror I noticed all my photos from 2017 & 2014 and most of my 2018 photos had disappeared off my network. Worse my two external portable hard drives, also seem to have lost the same folders. It all happened so fast I knew it could not possible be an accidental deletion and the two external hard drives where not connected to anything at the time. So how could they possibly have lost the data. Time to panic?
My "Air Gapped"Archive (one should be off-site but wasn't)It could have been BUT last year I had set up an “Air Gapped” Archive, on a Linux computer that was not attached to the internet (or my Local area network). My original plan was to rotate these on a three generation Son-Father- Grandfather cycle and store the grandfather off-site. However as I have only been updating the cycle every-six months (my original plan was every three months)using old computer hard drives. I only have two generations and they are both here. So I broke out the father copy and fired up my little Linux machine. It took a while but what a relief all the missing directories and photos where there. Next the son copy, all there as well, actually only up to mid December last year when I did the last archive backup. So I could be missing a few months, bummer! I unplug my archive drive and went and got one of my external backup drive, the one I knew I had used most recently, to my surprise all the missing folders where there and full of my recent photos.
The Backup Twins, The USB Harddrive I use to keep my ew and working files backedupIn disbelief I unplugged it (after a safe to remove step) and took it back to my desktop, no folders again. What is going on? I took the external drive to my laptop also connected to my LAN, the missing folders and photos could not be found. Back to my linux machine and they are there! What a mystery?
Actually I wasn’t so surprised I felt strongly that it was impossible that so many files on different media and places would instantly disappear. However at the moment I don’t understand why this happened. I’m just very glad I put some time into planning an archive system (with backups of the archive).
If you haven’t already got a backup system in place for your growing photo collection go and do it now. Then before to long plan how best to archive the collection!

Wednesday, May 16, 2018

Day Five … M is for Manual Mode

M mode stands for Manual which means the camera is expecting you to select and set the three exposure settings (Aperture, Shutter Speed & ISO). I’m sure if you are a first time camera owners this mode is very intimidating and if you just select it and started clicking away you may get a few good photos but you will also unfortunately plenty of under or over exposed photo (they are really only good for deletion, so hopefully they were not of an important occasion). The key is don’t panic, manual mode can be very powerful but requires some practice.

If you are happy with the concept that the correct exposure will be somewhere inside the Exposure Triangle, it is pretty easy to find a good exposure even without a light meter. If its a normally sunny day and you are shooting outside, firstly selection M (manual) on the mode dial, next set the Aperture to f8 (usually its a control dial on the front top right of your camera) and the Shutter speed to 100 (1/100th. sec) (usually the control dial on the top back right handside of the camera) and ISO to 200. The ISO is usually setting via the menu and/or a shortcut key on the thumbwheel style dial on the back of the camera. How do I know these settings? Well these are my favourite starting settings I’m taken a lot of photos with these. Other photographers might have different favourite settings to start.


Now press the shutter and  take the photo. Look on the back of the camera (assuming you have an LCD screen), does it seem a bit bright (Over Exposed) or dark (Under Exposed). If its over exposed like first photo appears, you need to let less light in. This could be achieved by using a higher fstop like F9 or F11, I used f9, or by using a faster speed 1/200, or as I choose in this case using a lower sensitivity (ISO 100).  Maybe this is a bit underexposed now. I’m still shooting just jpeg if I was shooting in RAW these difference would not matter I would be easily able to adjust the exposure in post processing. If the image is to dark you need more light and do the reverse adjustments to the settings. Change the setting(s) and take a second photo. Usually with a little practice you will nail the exposure by the second or third attempt and you can just discard the exposure tests, on a digital camera they cost nothing.

I was a bit shocked with my new Olympus OM-D E M10iii which actually recommended that my manual is exposure was not spot on and suggest F9 1/100 at ISO 100! Some other newer cameras will also highlight a suggestion or blink a setting that it would like to change. Smart little buggers they are.

Once you have practised this a little, preferable outside. You can move inside or into the shadows, but start with settings that would give your exposure more light to the sensor such as F6 or ISO 400. This is all great practise and should quickly let you find your preferred starting settings in manual. However there is an easier way, either take an Aperture priority or S Shutter priority test shot first and look at the settings the camera has chosen and start there.

The real power of manual mode, comes with a lot of practise and personal experience of really seeing the different intensity and quality of light. You will soon know the type of scenes you like to photograph and how you like them to look in the final print. There are dozen & dozens of ways shooting manual can help you get better photos, Such as “protecting” highlights or shadows (avoid “clipping” or “blow out”) restricting the exposure just to the subject … lots of creative ways. Too many to even to begin to discuss here. All I wanted to get across is don’t be scared of manual mode.

Tuesday, May 15, 2018

Day Four …cont. introducing the Exposure Triangle

If you are a new camera owner, you and have been following this series of blog posts by now you have realise that I’ve moved into the advance area of setting modes, without asking your to remember the exposure triangle or complex formula. You won’t totally escape, but it is just a brief overview here, and no test afterwards. This stuff is actually quiet straight forward, and it is much easier to understand when you apply it. I like this little infograph from Peta pixel because it includes a basic graphic representation of what happens with each setting.

Somewhere inside that triangle should be the perfect exposure for your scene but where?  Setting the mode to A or S lets the camera change one or both the other sides of the triangle and aims for that best exposure. Tomorrow we will explore M mode which simply means you will have to make all these exposure settings yourself.

The other thing you probably noticed is there is no I (for ISO) on the mode dial. I’m pretty certain this a throw back to the film days when the ISO was dependant on the film in your camera and you got to set that when the film was loaded but that was that and the ISO setting stayed until you changed film. Each type of film and its development process produce different sensitivities to light for that combination. The ISO was written on the outside of the film packet.  If you wanted a different sensitivity you had to buy a different film. These days on digital cameras the ISO can be changed electronically to let the sensor be more or less sensitive to light. In some cameras you get to change it to a single value, more recently most cameras let you set it to Auto within a given range and this lets the camera also flex the sensitivity.

There is a downside to higher ISO, in darker scenes which seem to demand higher ISO (and thus sensitivity) the darker shadows become subject to more noise (seen as a grain or an unpleasant spottiness). So staying with lower ISOs (setting your auto range 100 to 400) will probably be a good move when you are starting out, and you are not trying to take photos in dark rooms or alley ways. You can see am example of this noise in the shadow, of the flying duck photo in the earlier post, and look in the shadows behind the duck. You might have to click on the image to enlarge it. The noise is there because I had hit the maximum aperture of the kit lens I was using f4,6 and the low light meant the camera had to select ISO 5000 to get sufficient light at 1/2000th second. Normally I would have used a noise filter which would probably smooth this out nicely, but I am showing images straight from the camera (ok the duck photo where cropped) but otherwise not post processed

Day Four … S is for Shutter Priority

The S mode (Shutter Priority, TX on Canon cameras). Its very simple to use you set the shutter speed and ISO and the camera chooses the best aperture for you.

There is a lot of advice out on the net and in how to books about the importance of keeping shutter speeds fast enough to avoid camera shake. So setting the shutter speed sounds like a good option. It is also helpful if you need to you can “freeze” action by using and ultra fast shutter speed (like 1/2000th of a second), faster infact than a blink of an eye. While I did use this mode on older film cameras, I have seldom used this setting on any of my digital cameras (I’ll explain later below in this post). Like most cameras  my new Olympus OMD E M10iii does have an S mode and I was trying to think where I might use it and I settled on birds in flight (be warned that is a challenging subject to try yourself especially starting out, kids on a bike would be a better choice)

That was a bit easier said than done, It was a cold dull overcast morning that Melbourne gets a bad reputation for but aren’t really all that common. So there where not many birds out and about. I figured ducks would still be out, supposed this is the weather they like, so it was down to the lake.

Slow speed - stationary duckSlow speed moving duck -> blur
Fast speed flying duck -> sharpFast speed landing duck -> sharp

Despite starting to drizzle the duck did perform for me. He was sitting on the wooden rail on a small pier as I approached and a photo with the shutter speed set to a 50th of a second was almost ok (the image has a slight blur but the duck is recognizable despite the low light and slow speed). However as he took off, the next photo, also at 1/50th second is just a blur of feathers. He circled around which gave me time to change the shutter speed and catch him flying in and about to land at 1/2000th. of a second. Now most of the duck is in sharp focus (perhaps expect for him trimming his wing tip feathers). So Shutter Priority mainly helps in deciding weather you want blur or sharpness when you subject is moving. I probably won’t be using this mode to often, but it will help is some situation (if I have enough time to change the mode dial and the shutter speed before my subject has disappeared into the distance. Which it generally has.)  Thus you need to plan when using this mode before the action starts.

Monday, May 14, 2018

Day Three … A is also for Aperture Priority

I’m leaving the new camera owners to play around with their camera and find what it is capable off. Also the big surprise with my new Olympus OMD E M10iii was how good the jpeg was straight out of the camera (I noticed this because I wanted to just look at the images straight out of the camera and not processed in any way when I post examples in here). Purest might argue that the camera is actually doing some important post processing, and I can only agree with that. This only reinforces the idea that not recommending new camera owners only shoot raw and use manual mode, because it is likely  to leave them with discouraging results. Instead if you are a new camera owner, I think you are much better to stay playing in jpeg and have fun with your camera. The results will be good and you will still be able to improve as you understand your cameras strength (and perhaps weakness).

Unfortunately I couldn’t wait, to follow my own advice, I’m jumping ahead and switching over to the traditional modes, starting with Aperture Priority. On my new Olympus this is marked by a single A on the mode dial. On a Canon camera it is Av. I’m probably letting a secret out of the bag but a lot of photographers, the majority of those I know, (there is even a current survey over on Digital Photography School that also supports this) shoot in this A mode. In this mode you must select the Aperture (which controls how much light is let into the camera, and probably the ISO (the sensitivity of the sensor to that light) and the camera will work out the shutter speed to get an average exposure. This is the mode I usually use, and my excuse is I got into this habit back when I bought the original Pentax Spotmatic film camera many decades ago. It was a technical marvel at the time because you could set the aperture and then use the built in light meter (not a common feature at the time) to balance a needle horizontally in the screen by moving the shutter speed dial. It was magic and it got good results. Ok the shutter speed dial is gone on most cameras and the camera does the balancing itself without showing a big needle in the middle of the screen but it works much the same way.

The big advantage of using Aperture priority is its simple to use and pretty logical. In normal light you can pretty well always use a middle f-stop like f8, in darker situation you may want to open up and got to a lower fstop like f5.6 or lower. Let more light in when its darker, its logical.  With this basic knowledge and your camera in Aperture priority and you will normally be able to get a decent exposure in a wide variety of lighting conditions. Its not fail safe, there can be a downside if the camera chooses a very slow shutter speed (less that say 1/30 second) you may find a hand held camera is inevitably moved a little and your exposure gets a little blurred. Most modern cameras have some built in logic to avoid this and might blink the shutter speed in the viewfinder to highlight that the exposure is getting too slow and out of a safe range. Alternatively the camera may allow ISO to be adjusted within a nominated range by increasing the sensitivity (a higher ISO) in order to keep the shutter speed as fast (quick) as needed.

There can also be a creative advantage (or hinderance) in being able to choosing the Aperture, and that relates to how closely the camera focuses on your subject. In simple terms a tiny hole (like a pin hole camera or say f22) lets the lens focus on a wide range of distances onto the sensor. The bigger the size of the aperture the lower the range the lens can get in focus either side of whatever you choose to focus on. This phenomena is called Depth of Field. As the fstop gets smaller (the aperture is opened wider) and the depth of field gets narrower. Portrait photographer for example often use the feature to get the subject’s face in focus but blur the background to remove fussy distracting detail.  Alternative a Landscape photographer might want everything in focus so they will usually use a narrow aperture, f11 is a good starting point. There are many ways to use this sharpness or blurring creatively. The smaller the fstop of your lens the more power you get too blur but usually the more expensive the lens.

A narrow Aperture F11- Widest Depth of FieldAn Average Aperture F8- Medium Depth of FieldA Wide Aperture F5.6- Narrow Depth of Field

In the three photos above you can see this depth of field control at work. In all cases I focussed on the blade tip of my safety knife (also know as a Box cutter in the USA) which is opposite the 20cm (approx 8”) mark on my metal ruler. At f11 (a smaller aperture hole) you can easily read the markers out to 31cm (12”) mark before the marks and numbers start to become blurred. With the aperture set to f8 the scale starts to blur between 26 & 27 cm (between 10” & 11”) mark. At f5.6 the blurring starts at the 25cm (10”) mark. The important thing also to notice is that my pink plastic safety cutter is pretty much the same exposure in the three images because the camera automatically changed the shutter speed (and in my case also the ISO) to achieve this.

After becoming happy with Auto mode, I believe a new camera owner should next get the courage to try A mode (Aperture Priority). Moving to A will let resulting photos stay nicely exposed and occasionally you could start to pull off really good creative shots by  experimenting with and controlling depth of field. However don’t race out and buy an expensive fast lens (with a low f-stop number) just to become a bokeh master right now. Experiment with the lens(es) you have (a lot of kit lens will be f4 or f5.6, but they’ll be just fine to start).