Duplicate Duplicity

I’m getting very puzzled by the number of duplicates turning up each month. I have the Exclude duplicated clicked when I load anything via Picasa, but I have noticed that sometimes the other default to delete photos after loading in the after copying selection, is changed to leave card alone and so the picture might remain on the card for reloading another time (normally exclude duplicates then puts a cross against those images (so I avoid loading them twice)


So the problem remains why is it happening now? Also it seems to be far more common when loading from my Canon EOS DSLR than any other cameras., Further it is often only a few photos that are duplicated in a batch. So I’ve tried out loading with picasa and then overloading with a few other software and it seems that the most likely suspect is ligfhtroom. Perhaps it happens to be running when I insert a card but because it is so slow it only gets a chance to load a few images. However the problem is I don’t have lightroom set up to autoload images.

I’d be interested if anyone else has seen this lately.
So for the moment its back to checking as backup I my images and move them to the archive.

Taming duplicates

I have had the exclude duplicates item ticked on my picasa imports by default, for the longest time. This helps avoid a lot of duplicates turning up because some photos where left on a card or a phonephoto was uploaded via wifi more than once or via different services, However because I now have my photos flow across several computers and software packages it is surprising just how many duplicates do turn up. So I have also been using a neat little utility program from digital volcano called duplicate cleaner, which is design for all file types but does work very well with media files like photos, video and music. The biggest feature of this cleaner is that it can recognise the same file contents, even if the name is changed. If the file is a jpeg and has been post processed it will be considered as different. Also if the jpeg has been rescaled it will also be considered different. This is fine I probably want to keep these,

During and particularly as I reach the end of the month I move my photos from the collection points (laptop computers) onto a backup drive and it is at this point that it is most useful to run the duplicate cleaner (if you have a lot of files it can take a while …so go get a coffee). After this I then copy the backed up files to the archive drive and make my DVD/CD backup set of the archive (this is my one remaining traditional backup, everything else is now on portable hard drives). I don’t delete the photos on the collection computers until I have finished the backup –> duplicate clean –> Archive cycle. The very end of the month is the right time to do this.

A bit of a clean up

I’ve recently got a Western Digital My Book 1 terabyte USB drive, I like it, it was easy to install, I just had to reformat to use NTFS (It is distributed with FAT32 presumably for lowest level compatibility)and decided to move my photo archive to it. A terabyte is a lot of space, my collection only took up 207 GB, 125,820 photos. The trouble is I already knew my archive collection had been polluted a bit, with more than a few duplicates. Finding them by hand in such a large collection was not a task I was looking forward to.

Finding Duplicates with Picasa

After I’d moved my files, it took a while, I pointed picasa to watch the new folders on My Book drive, that took a while to “index” my photos. I thought I’d try out the picasa duplicate finder (its under Tools\Experimental\Show Duplicate files). I learnt I have duplicates in 683 folders. The bad news was unless the duplicates where in the same folder or adjacent folders (which very few where) you don’t get to see the duplicates. So I would suggest anyone still wanting to use this features, does so with extreme caution (after running a backup). Time for a rethink.

Duplicate Cleaner

Next I looked for a specialist duplicate file finder utility on the net, there are lots. I tried a few but the one I liked is freeware and called Duplicate Cleaner by Digital Volcano. I also does a good job with music collections. Basically it does what the name suggests it runs some fair deep searches, with intelligent ways to filter and direct the searches. Duplicates (or more of the same file) are group together and the tab displaying those groups has a useful image preview. What makes this more useful than the others I reviewed is it ability to match content, using Digital Volcano’s own MD5 hash code (a kind of file finger print) instead of just file names. The  bit I really liked best was that you have a choice to deleted to the recycle bin, moved into an archive elsewhere, or replaced with hardlinks (to the original). You can even save (& reload) the duplicates file list as a .csv format.
I now know that I had 934 photo duplicates. I first moved them to a new location, backed them up to DVD and deleted them. Daunting task over!