Hard Lessons on the Importance of Backups & Archiving.

In addition to the picasa database mix ups, which may or may not be related, I have suffered three external hard drive crashes, or malfunctions, in the past year, All of them contained photos but fortunately I didn't loose anything! I have been very regular with creating backups and maintaining at least 2 archives.

As all three of the drives were also holding my music library which iTunes was using at the time of the crashes AND there are so many hits on the net offering advice about recovering your iTunes library when you disc crashes (more often than not in iTunes 10). I started to get suspicious about iTunes 10 being actually responsible. No music was lost but I now rely on windows media play to listen to my music and just use iTunes 11 (which I really don't like) on an old machine to get podcasts and manage my old iPod Nano (which I still like)

Un-Jumbling Picasa

Non Destructive Photo Editors, have an achilles’ heal. Whilst they don’t change the original file (some may make a copy of it) they normal have to store the procedure that was undertaken and this is typically stored in a datafile somewhere. It might be a sidecar style file (like lightroom’s .xmp format) or a in the same directory (like picasa’s ini file) or even in a master database (lightroom’s catalogues). Because these files are normally hidden then it is possible to miss a corruption and damaged data could easily get formally rolled into your backups and archives.

In the process of moving around a lot of my photos I started to noticed that on at least two of my computers that Picasa occasionally displayed the wrong thumbnail. Clicking on a photo might reveal something else. The full view was the correct photo but the thumbnail didn’t match. Pressing the Refresh Thumbnails in the folder View did fix this locally. However the more I looked the worse the mix ups became, what appeared to be folders of new photos seemed to have some old photos scattered around. then the thumbnails would change! It looked bad. Yet I didn’t panic I know enough about picasa to know the original photos would be untouched. What was happening was some kind of mixup in the linking to the photos. But what about all my editing, tagging, and geocoding. Time to tread warily.
Then I updated to version 3.9, build 136.20, and my main photographic machine would not load picasa at all. Looking at the release notes didn’t give me any hint of what was going on The thumbnail mix up just got worse and worse. Time to get serious.

Fixing the Mix Up

The Google Picasa Help, recommends a fairly drastic way to Rebuild your Picasa Database, if you follow this approach you should definitely backup your photos first (yes they do recommend that). Yet I was still not convinced I needed such a drastic measure!

A much gentler explanation and procedure to follow appears in Picasa Tutorials , under the post/tip by Chris Guld, When your pictures get "All Messed UP" however her first approach no longer works (at least in picasa 3.9). The second approach might sound a bit scary but it is actually easy just make sure you have closed picasa and rename the folder C:\Users\User\AppData\Local\Google\Picasa2 (replace User with your user name) to something else (like …\Google\Picasa_corrupt).  Notice I suggest renaming the folder not deleting it, this might be vedry important (see saving Face below).

Now is the time to follow the instruction to uninstall picasa, download the latest version and reinstall it. When picasa restarts it will set about recreating the database. This can take a long time! So I did it in two steps for the initial run as prompted I just guided picasa to look through my pictures directory only. Later I used the File/Add Folder to Picasa... menu to identify my external hard drive folders and selected just the folders containing my archived photos. Rather than having the scan always I selected these to be scanned once and also checked the face detection on for these folders

Picasa.ini to the Rescue

Picasa stores most of the actions you undertake on your photos in a hidden file called picasa.ini which is stored in the same folder (sub-directory as the photos). Mostly this is to do with how the photos has been edited and what process/filter has been applied. Details of which photos you have starred and any album you have chosen to also include that photo within. Some of this information is also stored in teh central data files but the picasa.,ini file remain a secondary location and when picasa is re-installed and begins reading through your directories it can use this information to recreate you views in picasa as before.
Simple example of what’s in a picasa.ini file

[Picasa]
name=2013-04-07
category=Folders on Disk
date=41371.460810
[IMG_0999.JPG]
backuphash=278
crop=rect64(25c10987c527f872)
filters=crop64=1,25c10987c527f872;

As I had extensively used staring and albums as part of my organizational tasks as I load new photos into picasa I was keen not to loose this information, and it was preserved. Also the geotagging information was actuall stored embedded into the EXIF metadata so it was also preserved.

Saving Face

The one thing that may be lost in the picasa database rebuild process is the identity of which face can be recognized in which photo. This is a bit complex because the recognition that a photo contains a face is stored in the picasa.ini file, but like the thumbnails the actual contact name is stored in the main database. Further with the google+ changes the way these names are synced is a bit different. So the good news is in theory if you have sync to the web ticked on the tools/people manger tab (shown here) then your contact information will already be synced to gmails and/or google+. However what if they aren’t? Great Aunt Mable probably doesn’t have an email address anyway. That is why it was better to rename the picasa2 folder, because all your names can still be accessed in the file called contacts.xml. You just need to copy this back into the new folder once it is set up (ie soon after picasa start searching your computer for folders). As this process progressed I noted that under 3.8 build 136.20 where the person is identified by you, their name with their email address when defined, is now included under the new [contacts2] label in the picasa.ini file. So Maybe these hassle can be avoided in the future.

It  has actually taken me a few days (mainly taken up as picasa trawled my directories and photo storage location on external hard drives) but I am now confident I don’t have any more funnies in my photo collection. My confidence in picasa as a reliable non-destructive Photo Organizer has been restored.
So in the words of the Hitch Hikers Guide to the Galaxy Don’t Panic.

Trying some alternate geotagging software

I’ve been a bit quiet about geotagging lately, because I was been happy with my two favourite approaches to geotagging, using picasa/google earth &/or geosetter, also I am still a little worried about the dangers of over publising my location details. I do like being able to set the gps on my phone on, and collect a wayfile as I take my photos. Then I can later use this to match up with my photos, and have the embedded location in the EXIF metadata, the hassle here is getting the time matching, but once you figure out how the time shifting works it is nice to have your photos also organized by location.

Picasa (and google maps)

This is probably the preferred google way to do this. Rather than use the tools/geotag/geotag with google earth... you can just click on the red map tag on the left hand side of the individual photo or folder view screen. This pulls out the map dialogue, which is essentially a google map window. You must basically navigate to the photo location using an address, and you can not load your gps wayfile. However it is very easy to use and it immediately write the GPS coordinates (rather than the address) back into EXIF Metadata of the photo.

Lighroom 5 beta

Possibly like most Lightroom user, I did look at the Map module in version LR4 and had a little play but kind of dismissed it as nice but not what I wanted. Probably the biggest thing at the time was the fact that the location details where keep back in the main catalogue. It was a fair bit of work to ensure that the location detail got back into the EXIF data, also I has trouble loading and particularly got lost on the time shift dialogues. I suspect that not many lightroom fans have really delved However I have had a relook at the LR5 beta Map Module , and although I can’t see any major updates but I have found the gps tacklog load and matching does work well now. It also has a nice reverse geocoding feature that will determine the address and write that into the IPTC data. There is still the issue of the extra steps to write this back into the photos (or XMP sidecar files) however overall I it is pretty easy to use.

Photo Mechanic

  
I’m only trying out the demo version of photo mechanic, initially just to review photo load times, but I soon discovered it has neat getagging feature, being able to merge gps wayfiles (albeit I have only had success with .gpx files captured my google’s my tracks app on an android phone) and also accommodate timeshifts between the camera and gps device. Like picasa the details are written back into the embedded metadata in the original file (both jpeg & RAW). Photo Mechanic also have some very nice features for working with EXIF and IPTC metadata, including a IPTC stationery pad that lets you quickly update a lot of standard photo captioning. Also like picasa this is efficient and fast. This is a serious alternative to geosetter, that many keen photographers may already have.

Getting location into your photos is only the beginning, the real power comes with being able to use that as an additional means to organize and view your data. all of these options do give decent maps but you have to select the photos some other way first before you display them on a map.

Synchronise those watches (cameras)

The key to getting the allocation of geotags allocated automatically, when your GPS, or Smartphone is collecting the waypoints, is in synchronizing the clocks on the gps device and your camera.

Photo geotagging software, that merge external waypoints with photos do it via the photos time stamp. Some match to the closest waypoint, a few will let you interpolate a path between the way points. However this does rely on accurate time stamping on both devices. There are a few things that can conspired to get your devices out of sync.

1. Changing time zones, usually smartphones will automatically adjust as you travel into a new time some and the phone connects to a different network/carrier or even the same network in the new location. This is great but your camera won't!
2. Daylight Saving, this is a bit like changing time zone, where you smartphone, and computer, probably will but your camera may or may not. Further if you camera does change it might change on a different weekend to the gazetted times at the place you are staying .
3. Normal drift in the accuracy of embedded clocks, usually this is only measured in seconds but I just checked and my two computers are 56 second different, my phone a few seconds different again, my canon camera is 5minutes fast and my Pentax 2 minutes slow!! Ok last time I did check their synchronization was last October, in a different time zone.!

It really doesn't matter which is the "correct" time as long as they are all synchronized. A few second is probably not going to make any difference to reliable geotagging if you are on foot. However if you are cycling or drive between photo shots you might find some photos get misplaced.

Most of the geotagging software now does allow for time shifts between the camera and the gps device, but the method of correction is often very confusing.

Accurate Times.Is

Time.is, is a very simple web applet that gives you the accurate time at your location based on web connection to time servers that access an atomic clock. It updates at the beginning of each second. Importantly it uses nice big easy to read numbers so that you can read it at a distance while be setting your camera and phone.
Time in Monash:

Black Swans at the Beach

The bay was particularly calm today and the black swans were happy feeding at ricket's point.

Photographs of the Partial Eclipse

         A collection of photos as I photographed a projection of this morning’s partial solar eclipse from my backyard.

The Shadow Portrait Method, free light meter for digital cameras.

Looking back through my photos I notice I have a lot of “shadow self-portraits”, those shots where you take a picture of your shadow. However there is method in my madness. With a digital camera it is possible to get a reasonable idea of the correct exposure even without a light meter. Just point your camera at your shadow on the ground. Looking at the image on the back of your camera, can sometimes be difficult, because of the glare and strong reflection but even so you will usually be able to tell where the detail is in the image. If all you see is a very strong silhouette of the shadow and some detail in the ground you may be underexposed and you should do one or a combination of the following,open the aperture by perhaps an F stop, move to a lower f number, decease the shutter speed or raise the ISO a little, Alternatively you could increase The EV Value (exposure compensation). When detail is spread across the shadows and sunlit ground your exposure will probably be ok for an subject also illuminated by the same light (except if you are wanting to point at the sun or the sky!) If the sunlit ground looks washed out and the shadow not so strong and/or the shadow has the best detail you are on the overexposed side. So in this case you might close the aperture, move to a higher f number, decrease the shutter speed or lower the ISO a little. In this case you you want choose a negative EV. If you don’t trust the screen because of glare, the histograms for shadow portraits generally have two distinct bumps. If the bumps are over on the left there's a risk of under exposure, over on the right your probably find your photos are overexposed. Once your changed your setting taken another shadow portrait to check

Underexposed Shadow Detail Lost	Exposure OK	Over exposedLosing detail on the sunlit sand

Next time you are out on a sunny day, set your camera to M (for Manual) and point it down and at your shadow. Then start experimenting with a few or one of the above options. You should find you can find the correct exposure with as few a three or four test shots.

Waiting for the Watched Pot

Over the past few days I have been loading my photos (and Videos) from SD card onto my photographic laptop and timing how long it takes to load using various software packages. Its a lot like watching a pot to boil (yes the old adage is “A watched pot never boils”) it does but it takes a long time. At first the results didn’t make a lot of sense, The number of files or combined file size didn’t seem to relate to the time taken. Clearly Picasa was fastest and Lightroom 5 Beta (LR5b) was generally slowest, with Photo Mechanic (PM) and Lightroom 4.4 (LR4.4) occasionally taking the longest. This didn’t make a lot of sense because whilst a lot of folk seem to complain LR4.4 is slow, many have suggested that LR5b and PM are much faster. I suspect they are faster for JPEG and Cannon’s RAW format .CR2 but they don’t perform so well with .mov or pentax’s RAW format .PEF, then I realised there was a serious flaw in my testing. The cards I used in the two cameras were rated for different speeds, You can tell from the little number inside the circle that should be printed on the SD cards label. My Canon was loaded with a 10 (10mb/sec) but the Pentax was only a 6 (6MB/sec), clearly this has had the biggest effect on the results but like watching the pot I’ve lost interest in trying to be scientific and fair. I just need to only use SD cards rated 10 and load my photos onto my computer using picasa. Enough time has been wasted already!

Number & Size	camera	Type of files	picasa	PM	LR4.4	LR5b
52(640mb)	canon	.jpg .cr2	25	37
215(1.7GB)	pentax	.jpg .pef	32	75	80	73
11(1.13gb)	Canon	.mov	35	36	75	111
134(1.4GB)	pentax	.jpg .pef	42	54	104
184(2.4GB)	pentax	.jpg .pef	57	100	102	129
8(889mb)	Canon	.mov	28	53	86	103
13(286)	Pentax	.jpg .pef	6	22		19
72(920mb)	pentax	.jpg .pef	18		58.9	73
3(69mb)	Canon	.mov	7		15	22

Saving face, after a picasa database corruption

I spent a lot of last weekend carefully timing how long it took various programs to load my photos from cards. Picasa was the outstanding performer by now it has let me down badly. It refuses to start on my main photographic computer at all just giving me this error message and if I say yes, reporting an error with that specific file.

I have picasa on several other computers and two of them, both updated last week are showing occasional signs of corrupted data base problems (ie they often display the wrong thumbnails for recently uploaded photos. This has a frequently report problem since 3.8 but something has clearly made this worse on my computers. I suspect the 3.9 update. The recommended way to fix a corrupted database in picasa is to firstly delete picasa and then reinstalled the software a fresh. Yes that’s what it recommends!

This is fine for most features because the editing processes for your photos are actually stored in the picasa.ini files that are located in the same directory as the photos. the central databases just storing a thumbnail of the photo and its location, EXCEPT for some details if you are using face detection feature and any albums you have used. The flag that a photo might contain a face is stored in picasa.ini file BUT the detail of who is who (ie the names you have given the face are actually stored in . I suggest you might want to read the well prepared tutorial "Saving Faces When you Re-Install Picasa"

As I am in the middle of rethinking how and where I storing my working copies and archives I figure now is a good time to bit the bullet and re-installed picasa a fresh and get it to re-index only the photos and directories I want. I guess if it all goods well you probably won’t hear more in here BUT I am a little worried its going to be more complex than that! Things didn’t great off to a great start, when I tried to report why I was uninstalling picasa, I repeated got the following error!

See also my Un-Jumbling picasa update

PhotoFriday :: Temperature

For PhotoFriday‘s topic Temperature

Winter is coming, and the lighting can be dramatic

Winter is approaching and the skies are getting much more animated, In particular the glimpse of strong sunlight between dark clouds so often bringing the landscape alive. Its a good time of year to always have your camera with you and not be afraid to stop and take in the wonderful lighting effects and get a few “special” shots. Both of these shots, where taken by the side of the road when I noticed a fleeting moment of great lighting. Both where shot without filters but I did use the Adjustment brush in lightroom to lift the highlights a little. The Panorama below was created with autostitch using 8 “portrait” orientated overlapping photos.

Dark Skys

Autostiched panorama created from 6 photos.

PhotoFriday :: Remembering the Little Things

 Some older photographers may get nostalgic about the "good old days" of film, but I remember when working 1:1 meant using a magnifying glass on contact prints or the slides and negatives themselves. We have made progress!

For PhotoFriday‘s topic Little Things

Photo Importing Times

I decided to try out the loading times for some common photo management software (ie packages I have and am familiar with from my many mistresses project). All photos where loaded from the same memory card, a Lexar Platinum 4GB with a 4 speed rating. There where 24 images ( 12 photos of clouds with both jpeg & raw versions saved) with a combined file size of 225MB, taken over two days. All where loaded onto an older computer (running windows Vista) which I knew would be slower but that would give more a better chance to more accurately record and compare load times. Finally the loads followed my normal procedure to upload the photos and delete them from the card (ie I had made a copy of the cards contents and restored them between each test). In lightroom I used no extras, no keyword/exif updates and minimal preview render.

Package	Load from SD card	Smart previews	Comments
Picasa	8 secs
System Copy	11 secs		Drag & drop from card
System Move	14 secs		Drag & drop from card, set to move rather than copy.
Windows Live Photo Gallery	13 secs		* Did not load some RAW files
Synkron	17secs		** Does not include setup time
Lightroom LR5b	18 secs	27 secs	Does not catalogue matching JPEG but did load them
Photoshop Element PSE	-		knew better than to try it!

(**) I have been experimenting with using file synchronization to move my photos around my LAN and I have added using synkron as a photo import function. So I had to set up a customized synchronization task to move the photos to a folder based on the date photographed and verify the copy before deleting it

I was actually a bit shocked by these times. Whilst I already knew picasa would be the fastest I didn't realize it was faster than a file copy at system level? I now have a lingering doubt that picasa may not verify the copy before deleting the original on the card, However you do see the thumbnails of each image being created from the folder on the hard disk, which does give immediate feedback and some confidence that the file load went ok. Windows Live Photo Gallery was also fast, again as I expected, however it was disturbing that it had chosen not to load some RAW files(*) while it did load others, and there was no message about this at the import time.

All packages correctly loaded the data into two separate folders based on date taken. However there was quiet a variety in the actual names, so they all loaded into different folders in practise! This could become very frustrating.

My current preferred procedure for loading into lightroom or photoshop elements for that matter, is to only import the photos from the hard disk location after they are loaded via picassa (ie from the two directories Picasa had set up). So the following table has two times, because there are two folders, 14 photos in the first sub-directory and 10 in the second)

Package	Import From folder	Smart previews
Lr4.4	4 secs 3 secs
lr5b	3 secs 2 secs	16 secs 15 secs
PSE	6 secs 4 secs

These results are consistent with the general discussion about the new lightroom 5beta, it is somewhat faster than lightroom 4.4. The interesting outcome is that excluding the time lost swapping packages and selecting two different folders. It is clearly faster in elapsed load time to import from the cards into picasa (8 seconds) then import into lightroom (4.4->4+3=7 total 8+7=15seconds, or 5b->3+2=5 Total 8+5=13seconds). The creation of smart previews actually took longer, which is a bit of a puzzle. For a few photos this difference may not amount to much but for a full card or several full cards the time saved will be significant and measured in minutes! I am not surprised that the photoshop element was the slowest, my version 7 organiser is showing its age, and hopefully the organiser has been performance tweaked in more recent versions, I just keep PSE around for the few occasions I want advance manipulation and perhaps layers, although I usually prefer to use Corel Photo Paint in such situations.

So really I have just confirmed what I had already chosen. Picasa is my favoured mistress to load photos from camera cards and to do the preliminary organisation and weeding of my photos. Then I later import only the ones I want to keep into lightroom, my favoured mistress to finesses my raw photos, but from the date based folders created by picasa. Thus the story of more than one mistress continues … ...

Numbered Stacking, is great BUT…

I often take a number photos, in a specific set, usually bracketed groups of +/- EV settings, or adjacent & overlapping images for panorama stitching, etc. These can clutter and overwhelm your grid view or film strip. Light room has a great feature called stacking (it has been around since LR3 days) that lets you, hide the rest of the series behind the first image. It works a lot like having a set of photos underneath the first photo (normally the first photo taken, although it is possible to select the photo you want to be on top of the stack. The photo stack are identified in the library model and grid view by two vertical lines on each side of the photo. Clicking on either line will expand the stack, click on the vertical bar again with close the stack, The short cuts for this is S to open the stack (and once the stack is open press S again to close it). There are a few more stacking options under the menu item Photo/Stacking.I use this feature a lot and couldn't help noticing a very nice new way the stacks are displayed in LR5b (that's lightroom 5 beta). The stacks now have a small stack icon displayed (the icon is simple overlapping rectangles with the number inside the inner rectangle) at the upper left which shows you the number of photos in the stack. This is much more obvious than the two vertical lines (which are still there) and because I know what it is I usually leave the stacks compact. The problem at the moment is if I am showing others my recently loaded photos, they are immediately inquisitive and expand the stacks, or  want me to expand each stack. So I got to wondering was it possible to toggle off the stack numbering (and I can't find a way to do it) Then I realised it would be wonderful to be able to flag these boxes with colour (or a different shapes) to differentiate the type of stacked images, (eg HDR set, stitched panorama, group photos). Oh that’s right lightroom doesn’t yet support any form of photo merger (yet!)!
So here are a couple of interim solutions that work fine for me.

1. I must remember only show others my lightroom collections (not original folders) so they only get distracted by what I want to show them.
2. I have started to use the colour flagging to identify what are bracketed EV ranges (red) versus overlapping images for stitch panoramas (green)

Lightening, Overcast & Humid, No Glimpses of the Moon

 

I did get up especially and spent nearly an hour with my camera pointed skyward, hoping for just a glimpse of the lunar eclipse, but the skies here in Jakarta remained heavilyovercast. In fact the clouds where so low they where well lit my the city lights. The constant thunder and lightening did backlight the clouds and give hints of a few gaps, but the slightest trace of the moon, or the shadow of the earth on it, never materialized. Adding to the challenge was the extreme humidity, so everything on the camera fogged up. Even after half an hour to acclimatize, I found I still need to have my cleaning cloth wrapped around the camera.

I did manage an available light photograph of the fountain, beside which I had set up my tripod.

Cross Town Traffic

The traffic in Jakarta is a really intense, albeit very slow experience, but it is a challenge to show in still photos. Such disorganised chaos that actually still moves is pretty amazing, but oh so tedious if you are in a hurry. To help those stressed out many street corners now host massive flat screens continually showing commercials!