Ransomware & other ways to loose your photo collection

The Wanna.cry hullaballoo brings to light the vulnerability of keeping your photo collection on-line, on a computer or phone but in one place. This is running a big risk, do something about it now.

Avoiding most Ransomware, is not difficult, its just needs you to make small changes in normal behavior and regularly be wear of or do a few things. Be careful about clicking on links in unsolicited emails and phone message that have the characteristics of a phishing exploit. Be aware of identity fraud and becoming a little more private, be careful and aware in what you share. Even basic snapshots can hide a lot of useful information to a stalker (eg location) or hacker. A good example is the common trend for people to post their boarding passes from the airport gate on Instagram or Facebook which gave hackers enough personal information to gain free entry into accumulated frequent flyer points which they immediately stole. The most important protection is keeping your security/protection measures up to date. Which means installing rather than ignoring those security updates. The final and most important way to not be caught out with ransom wear is to keep an up to date backup, under your control, then the ransom on on a web site/computer becomes meaningless.

However ransomware, virus or worms are not your only risks, Equipment loss damage of malfunction is a much higher possibility and in some cases approaching a certainty. In the case of mechanical spinning harddisk it is a certainty that they will eventually fail (fortunately generally their life will be about 20 plus years). Equipment particular smaller portable items are also prone to get damaged or lost. Mobile phone really don’t like getting wet or dropped. There contents sometimes survive, particularly if have them on a microSD card but I hear many sad stories  of photos lost this way. Making sure your phones get backed up properly is the guaranteed way to avoid the risk. Sure you can sometimes recover files from damage harddrives and phones but it can be very expensive, its much easily to go get new equipment (your going top have to spend that money anyway and pick up an reload from your backup.

 

So what is proper backup. There is lots of advice on the net but the best approach is the 3-2-1 method (see the basics in the David Bergman Video above). Have three copies (yes three) on two media/devices (eg disk, DVD etc) and one copy off-site (eg on-line, bank vault, a friend house across town). There is a lot of options in how you actual do these steps and set up something you can easily manage. It may involve some extra expense (like buying a second hard drive subscribing to an on-line backup service) but remember you are responsible and the should take control of the steps (ie don’t expect a free social media service to do anything special about “your” photos.)

Go start that 3-2-1 backup today, don’t delay it could be too late!

Backing up the actual photo files is also only part of the issue. There are important aspects like how you rate the images and organize them, and in the case of non-destructive editors how the image has been manipulated. Some of this information can be written back into the header of Jpeg files but most photo processing software today stores this separate to the software. In the case of lightroom it is all stored in database  its calls the catalogue. other software such as picasa store this data in a special ini file in each directory containing photos. The original adobe bridge software stored the information in a readable ascii companion file (.xmp) normally known as a sidecar file. Whilst these are not standard, and adobe itself has muddied the waters by changing some format so you need to know the ,they have been around a long while and many other software packages can at least read the metadata and ratings in these files. Some packages (like Corel’s Aftershot Pro and XnView) can also write the basic .xmp files. You need to look at the help (or manual) for the software you used and work out where these files are and how copies can be included in your normal back up set. While Lightroom does itself keep a separate backup of its catalogue it is likely to be on the same computer as the software and photo library. So it is important to include this in the 3-2-1 steps.

If you’re a regulkar reader of this blog, you will know I keep harping on the issue that backups are different to archives, but while they are also copies they should be organized in such a way that photos can be easily found (that metadata discussed above) and there is also the even bigger issue of what format have the best longevity.

Backup your software and operating system by creating a recovery disk (eg in Windows 10, you get the opportunity to create some recover media from which you can reboot your computer) in the case of hardware or software problems. Its fairly large so you will probably need a 16 GB USB drive. I made mine (for my three windows 10 computer) in fluro green) so they look different to any other USB memory key I have. Luck I made the effort because my main desktop reported an update error, and after clearing the upload and reloading the security patched it still seemed unhappy and after the next shutdown it refused to start. Rather than panic I got out the recovery key and rebooted from the USB. Five minutes later I had my desktop back. So make that recover media now. Another item you must remember to backup is the software you use to view your photos plus the registration Keys that you got when you purchased the software. Don’t assumed that you will be able to always download older version of the software. I know corel, adobe and microsoft are not making what they consider superseded available for download. So Search out that install package and include it in your photo backup. Digital photos are binary files and need to be interpreted and rendered to suit your output device, by software. The big risk with proprietary RAW photo format (which many on-line photographers encourage you to use) is they may not be supported in future software. Jpeg is an accepted public standard and so ubiquitous that it is unlike to not be viewable but .DNG, .CR2 .Nef etc might be a little like having a beta Max tape but no player.

Regularly make your best/favorite/precious photos in photobooks, or at least 4” by 6” (10 by 15cm) prints. This is really a physical variation on the backup (in another media) but it does work. Think of all those old shoe boxes with photos from your parents, that have survived the years. Unfortunately many get trashed in post death cleanups. They are controlled by you but can still be more directly/privately shared by you to just those you want to see them.

“Privacy is becoming a privilege” … Paul Thurrott (Windows weekly podcast)

A little amazed at how well flickr is automatically adding tags

I’ve been keeping up the photo a day. They are not masterpieces just different takes of everyday life mostly. I’m not sure why I choose flickr to post them as many are ordinary just me trying out different pictorial and composition aspects (rather than photos the “pixel peepers” may like). What I have noticed is how improved flickr has become at analysing my images and adding automatic tags. The only tag that was wrong today was indoor.

(I can’t imagine an indoor barbeque?)

The Importance of a Catalogue to your Digital Photo Archive

A shoe box or album full of photos is easy to flick through and you can quick see both its condition and content. Digital files however have limited visibility, especially a collection of digital photos. The Photo file will be in binary and unless you have specific software to decode it it will just be symbols and numbers at best. One exception is the jpeg format which is so widely used that most software, browsers, computer and smart phone apps with recognise and decently render it. Other formats and in particular the camera RAW formats will always require social software (and given that these formats are proprietary and there is a risk of them becoming obsolete) may not be guaranteed to be available on different hardware and/or in the future. Further It is unlikely that scanning the file names from the camera (like IMG_0007.CR2) will be very meaningful. You need a reader that understand the format of that data and can render it as an image. A decent catalogue (or Index) of you photos need both to be able to easily view you photos but it also benefit from some organization that makes the images quicker to find

So how do you figure out what is there. There are two common solution,

1. The first is Thumbnails, normally postage stamp size low resolution version that give you a peek at what is in the photo.
2. The second is to use metadata (data about the context of the photo itself).

There are lots of decent (and free or inexpensive thumbnail programs) like Picasa XnView, Photo Mechanics, the default folders view in windows even lets you select a small, medium or Larger icon view, which are thumbnails. Some can handle RAW formats but other may not (or rely on updating a codex for that format). Other software like Aftershoot and OnOne 10 have a browse mode so you don’t have to “import” photos first, you just get presented with a grid of thumbnails. These are fast and a joy to use compared with opening up directories of images in say Lightroom. Having a good thumbnail facility that is fast and easy to use is the first additional need for a good archive.

Metadata, (data about the photo and it’s contents) is a little more complex but it should not be. There are two standards EXIF and IPTC metadata formats. The EXIF data is really mainly about the camera and image and has been well addressed by camera and phone manufacturers (although they may not record all the possible metadata). This EXIF metadata is included with the standard jpeg data format so the metadata travels with the image embedded in it. The IPTC Interchange Model is a way to record information about who created the photo and how it is licenced for distribution and publication (it was set up for exchange of information, and images between newspapers  and is not a big deal in a personal photo archive). This information can be, but seledom is, embedded in jpeg files. Most software that works with photo will let you display both forms of this metadata. If you are having trouble reading the metadata Phil Harvey has written a wonderful open source program called ExifTool, which can interact with metadata in many formats (eg, embedded in the jpeg or stored in an .xmp sidecar file)

Whilst you can see a scratch of water stain on an old photo. It is tricker to judge damage too (corruption of a digital file) because in its raw binary digital format it if not easily comprehended. Even worse even a little damage may render the digital image unusable. There are technique to ensure digital files are not corrupted, most notable the checksums and Hash function (they are not hash tags). Of these MD5 is the most used for photos and is usually the technique of choice applied for detecting duplicates. Whilst I can see some software uses checksums (see picasa.ini of right) they do not appear to be MD5 numbers. Adobe use two different IDs which look a little like MD5 hash values, the first is a unique image ID created when The image is scanned by lightroom (or other adobe utilities) and the seconds is an instance ID and is updated whenever the files is accesses and parametrically adjusted (ie the .XMP file and/or catalogue are altered). I can not fnd any definitive documentation on them So I’m just going to assume they are yet another propriety secret and thus I doubt I will research them any further.

Fortunately there are lots of well tried and tested MD5 utility and programs, many free to download. What I discovered is the best way to create these hash table is not one for each file but with one file for each sub-directory. I already had a program called total commander (and alternative to windows explorer) which does this. Further you just need to click on that file and then Total commander will scan the files mentioned and report back if the hash value matches. Yes I did a test truncating a photo and the checked reported a hash value difference. There are a number of utilities so I assume you will have no problems finding a similar one.

I have been using this approach for the past 6 months it very simple and I just create the Hash Table for each directory when I do my monthly backup checks. I have been reprocessing some older directories as I move to a hard disk based archive. Running the hash function did Identified 2 photos that had read problems, luckily I had another copy of those which was fine. I am now slowly adding hash table to all my archived photo folders.

I’m now even more certain you need to also keep a checksum or hash value, as well as an easily accessed thumbnails and standard metadata utilities in addition to your archived files. Copies of the applications used to created these views and information are an important part of your catalogue and should be stored with your archived photos.

Not all is harmony in the XMP World

Whilst many software applications can share metadata via XMP files, the situation is far from a recognized standard.  XMP is an extensible XML style format, which means it can be customised.  Applications that read this data, need not be able to understand everything.  They just need to honour (and not modified) any such data.  Adobe itself use the XMP the file to store details of proprietary Post Processing methods within Lightroom, These methods are unlikely to be known by the software, and that software should not be expected to reproduced the exact Lightroom process.  Corel’s Aftershot Pro also uses its own proprietary XMP format for its processing, but also offers the opportunity to save standard XMP files by which it means just the metadata.

Is in the area of metadata that there does seem to be good agreement between packages.  Specifically key wording, star ranking and the colour flags/patches or classifications do seem to be reliably exchanged between programmes.  Most of the EXIF data seems to be so exchanged, when exception might be camera settings and lens identifies (usually of read from RAW files) that don't always make it into XMP files. Some some software does not acknowledge IPTC metadata.

Finally I don't expect to be told what isn't read from and XMP file.  Anything that isn't under stored get ignored.  So for example you may have set the Likes and Dislikes in OnOne's perfect photo suite, but they will not be interpreted as Flag as Pick or Set to Reject  in lightroom so they just don’t show up.

Bad Flickr …not bad panda

For some reason I believed that flickr had stopped stripping the EXIF data out of images when they are shared. This might not sound like a big issue but it is for those that rely on the image having an embedded copyright or creative commons entry in the IPTC and EXIF metadata. This in theory means that anyone that “finds” an image on the net should be able to trace the image back to its rightful owner. The trouble begins when various countries are enacting so called “instagram acts”, which allow “orphaned” photos to be freely used without and copyright or creative commons restrictions. Stripping the meta data away when and image is displayed effectively makes the image an orphan, if the user doesn’t want to take any further effort to find its owner.

I was experimenting with some of the sharing and social network integration in Irista, and happened to notice when irista bought images back from flickr the metadata was stripped. This surprise me because valid meta data is a key check that irista does on upload.

Flickr is only one of the popular services doing it. The only service I have found not doing it is google+

Looks like it is back to always including a visible watermark on flickr, or more probably not uploading important images there anymore.

Virtual Albums

One aspect of digital photo management that I like is the ability to create virtual albums (collection in lightroom).  These are not duplicate copies of the photos just different views that show only the photos selected without having to move to the folder/directory in which it is stored.  The photos remain in the original folder on the computer, there is just an additional link entry in your album to find that photo and display it.  It is the perfect way to store and show only a best photos or photos on a given theme. They are a great way to organise your photo collection and probably deserves a more detail post in the future.

The downside using an album structure in current software is that it can be very hard to transfer this structure to other computers and/or software.  Each package has the unique way to maintain the album contained within its code.  A few packages will let you exports and upload the contents of albums but fall short of being able to export the album structure,    For example Lightroom does allow their collections to be exported as lightroom catalogues, then reimport this catalogue into lightroom on a different computer.  The process is simple enough but is better described by Steve’s Digicam post. Another examples is the original Picasa and Picasa Web Albums (the on-line side, which is now google+ photos) which lets you maintain matching Albums on-line as on your computer and synch the two.

Use Album Keywords as Metadata

A neat way I have to be able to exchange the the organization you have established with your albums and then be able to exchange this between packages or onto online services that support the Album concept, is to code all the contents of the album (or collection) with a unique Keyword, tag or Hashtag. I was for a time using the ~collection_  prefix to identify this organization metadata by I have abbreviated this to ~alb_ (for Album). Of a  photo or batch of photos is transfered it is usually easy to search for the new metadata, Tags or Hashtags and create a new album or similar grouping from the search results.

To finish here is a great post by +Will James, on how to reorganize your albums in google+/picasa web albums.

Now you see it, now you don’t

its a while since I’ve mentioned flickr, in fact i really haven’t been posting to it much. Not because of the new format (I’m not really a fan but its much the same as other offerings) its all about “stripping”. They are still stripping the EXIF & IPTC metadata (including details or copyright or creative commons) and the photo ownership details) when the images are downloaded. This unfortunately creates what are known as orphan works (photos that in theory you can not establish the photos ownership). The problem being that in some countries it would appear re-publishing such photos, without the copyright holders permission is acceptable providing the re-publisher acted in good faith in searching for the photos owner, even though that was unsuccessful! See the legal loops holes opening up here!!!

 

Flickr isn’t the only social media service to do this but they are the one I would prefer to use.

This actually posses a big problem for creative people who want to share their work, even using creative commons. For example I don’t mind other using my images providing they acknowledge me, and that they don’t use my images for their own profit. A lot of people have contacted me and all is sweat, Unfortunately using google image search or TinEye I know some of my images have been used all over the place without my permission. So I have reluctantly taken the approach of adding a visible watermark onto my flickr posts. In reality this is not going to stop unauthorised reposting, or help me track down those just stealing the photos. To do that I should just stop posting to flickr (or most other social web sites). I hope it will just make those thinking of using eth photo think twice before downloading it. Flick actually has a good blogging link feature that includes a link back to the original flickr image (which does have the EXIF data intact).

If you want to use one of my images for a blog, or brochure for your organization, or any other reasonable use and you don’t like watermarks just send me an emails

Searching a photo database …

earching for a specific photo is not such an easy task on a computer right at the moment. The way we recognise things visually is still a challenge for any single computer application to match. Certainly we know a lot about how our brain & visual cortex deconstruct the light that reaches our eyes and how it recognizes shapes and edges and builds a tonal representation, and recognizes colour and saturation. Individually various pattern recognition systems can reproduces one of these tasks at a time today. One on the “holy grails” of computer vision within the artificial intelligence community has been in recognizing objects, potentially those to be manipulated by automated robotic systems, and there are been some successes in very limited spheres of operation. The fact that most smart phones come with a camera has been a great impetus to broader applications of computer image recognition, and already in a few specific fields applications are becoming available, like google’s goggles, which can reasonably accurately recognition a lot of consumer products snapped by smartphones.. But there is still a fair way to go before we have generalized recognition systems that might be really useful in searching through large photos collections and helping organise them are well as any human might. Although I do suspect suspect competition if becoming fierce to solve the more general recognition,  some interesting current development are everpix,, IQ Engines soon to be included in Flickr and here is a brief description of How Google's Image Recognition Works, in Google Drive & Google+. Yet right now

Ok back to the more pragmatic, what is available today.  Finding photos usually relised on a text based search of keywords associated with the photo. I know a lot will say just use lightroom, it manages your collections. I do use lightroom, but it is not really so good at search. It does have keywording and relies on the user to adequately keyword and tag all the photos. The main tool is library/find, which basically gives you a totally text based filter that can browes a subset your catalogue by keywords.attritubes, ratings and EXIF metedata. This all requires a lot of upkeep to keep this useful and there are some good tools to simplify updating the metadata but it remains a largely manual process. A lot of folk overlook the power of sort (again it has filter style dialogue) to help locate and organise (for example it it easy to sort by landscape or portrait orientation or by aspect a ratio if you know you used a square crop for example. the real power is hidden in smart collections, which are dynamically updated  as you classify your collections. LR 4 and above does have a map feature, and it will display any photos with already embedded GPS coordinates or let you drag photos from the filmstrip onto a specific map location, the search capability is just my map window and limited filters such as tagged and untagged photos.

One little package that surprised me with its ability to search was XnView, (which I have as my image scan, sort and keyword tool for my USB darkroom portable apps key (ie a backup when I’m travelling and don't have access to my own computer and picasa). I haven’t written much about this package but it is a gem, Basically it is a thumbnailer, It just creates a database of thumbnails and loads that with all the information it can glean from the photos into its database. It does have a few different ways to look at and organize the thumbnails, Firstly somewhat like the Lightroom filters, it has a single but just about as comprehensive Tools/Search…feature. It also uses a special type of keywording that it called categories which can be set up like a tree structures/classification system (and it comes with some good basic starting categories. The View tabs also have various viewing filters and sorting options which can be useful in sub-setting what you see.

I must admit picasa is my go to application when I am searching through my substantial photo archive. For a few reasons, Firstly it is tolerably fast (lightroom isn’t), second it has face recognition, thirdly because I use its default file naming as I load photos my collection is nicely organised in chronological order. Whilst I seldom use its places features to tag location i do use this feature to view and find photos from a given location. Its search box is in the classic google style although it does just have a text search functionality looking through filenames. folders and album names, keywords(tags), captions place names, and EXIF metadata. It is not really as comprehensive as the previous two but it simple and fast and the screen view changes according to what is available. The filters are there too but are different button/icons that further control what is displayed. While there are no “smart” collections,  people can be automatically grouped by the facial recognition, starred photos are continually updated to a special album, as are recent uploaded photos,

So my conclusion is while the future promises a lot, the present delivers just a enough.(if you are happy to maintain your own keywords)

This post is part one of my rethinking the longer term focus.

5 steps to avoid sharing your on-line photos with the world!

There are many services that offer to share our photos on-line, and most, specifically the social media places, make it very easy but have defaults that mean your photos are being shared with anyone with net access. Perhaps you just wanted to share your photos with a few friends of family members. To really scare yourself, try finding some of your own images on-line using google’s neat reverse image search. To do that just drag and drop a photo you want to check onto the little camera icon on the google’s images search screen.

Ok now your scared! Time to pause and check whether that photo also lets others know where you live. You can find the extra information buried in your on-line photo by copying its URL to metapicz

So how do you fix it so your images and the hidden information they contain are not being shared with everyone in the future? Here are 5 steps you can take.

1) Make sure you know and set appropriately the default sharing setting when you upload photos. Most of the specialist photosharing sites like flickr, google+, photobucket, smugmug etc have a multi-level security/sharing system for just family, friends or then the public. Google+ have the added organizational control of just sharing with your selected circles.  Some cloud based photo services like Skydrive, and Drop Box also have multi-level sharing security and the private side normally involves sending those you wish to share with, a secure link (ie it has an access password/switch built into the link). A big limitation of these systems is in the  photo uploaders they provide, which often have public set as their sharing default. So you can unknowingly make your photo available to everyone. So it is in your interest to set the default to private and only set those you intend to share as public on an individual photo basis.

2) Be careful that you don’t share links to otherwise private photo, by reposting to public sites (like you blog or a social network site like twitter). in those systems that do allow you to share your images on another service such as facebook or twitter, Often they use some variant of OAuth to establish the authority to post on the attached site, unfortunately they seldom inherit the same privacy setting from your original post and the photo is posted as public. Generally if there is no mention of sharing security when you do the authorization there will be none (eg Sharing to twitter from flickr, even if the photo is private on flickr once it is shared to twitter it will be publically viewable on twitter)

3) If you don’t want strangers to find out where you live, work, go to school. Consider turning off the GPS in your phone, or finding the setting in your photo uploader to remove geotags/geolocation data. Flickr has a nice geo fence feature that allows you to supress showing location if your are close to sensitivity place (eg home, school etc). Lightroom has a setting to remove any geolocation data when posting an image via their Export or Publishing Services. You can also control the types of metadata exported.

4) Remove the photos from google search if necessary. I haven’t been a fan of the google’s official forums because they are often hard to figure out the context of the advice being given and even what the FAQs are actually asking. My impression has been that google staff must not actively moderate their forums or perhaps even read them (maybe I’m being a bit unkind here). However the Picasa Resources site does provide some good discussion on key FAQs about picasa and its advice on How to remove your images from google search gives some good advice on what to do, albeit quite technical advice. The photos don’t have to have been shared via picasa.

5) Don’t use Facebook (and probably Instagram), for any photo you don’t want to share with the world because someone else liking or tagging your photo will start sharing that with all their friends and so on. These access links can spread quickly and soon many folk you don’t know will be able to see and use your photos. Also the facebook TOS (terms of service), means they assume ownership of anything you post. I don’t have a facebook account, and I don’t want to be accused of facebook bashing but I understand in the past it wasn’t so easy or obvious how to remove and permanently delete photos, I gather facebook has yielded to pressure and here is a link to Steve’s Digicam with instruction on deleting facebook photos and albums.

So don’t panic, make sure you understand the sharing security on your favourite photo service and set the defaults appropriately. It is certainly not wrong to share your photos with others. It might just be wise to do little bit of checking right now for any existing images you don’t want shared publically.

Blimey, …they do that? ..aka(time for a manifesto).

I had heard the story about a new law, known as the instagram act, in the UK making “orphaned” photos free from normal copyright restrictions on republishing. So how does a photo become orphaned? Most jpegs, the most common format for posting photos on the web, have important detail about which camera, its setting and who took.owns the photos embedded in the photo itself, These are the EXIF and IPTC metadata and because they are embedded within the photo they travle with it when it is copied. Whilst some older programs may not handle them correctly, programs like picasa and lightroom and many others help you expand the details stored here. However then when I started to investigate some of the recent social photo sites I came face to face with the fact that most of the social sites where stripping out the metadata. In fact of the sites I have used only google+ is not altering the metadata. I must admit I was pretty shocked that Flickr was one of the worst offenders.
Right at the moment, posting your photos on social web sites, may not protect your rights much despite anything they might suggest in their terms of service. Add to this this “overlooking” of creative commons licencing on google+ and the web has become a less secure place to “share” your creations. Fortunately there is some good guidance being prepared.
IPTC Photo Metadata Working Group have prepared five guiding principles as our "Embedded Metadata Manifesto":

1. Metadata is essential to describe, identify and track digital media and should be applied to all media items which are exchanged as files or by other means such as data streams.
2. Media file formats should provide the means to embed metadata in ways that can be read and handled by different software systems.
3. Metadata fields, their semantics (including labels on the user interface) and values, should not be changed across metadata formats.
4. Copyright management information metadata must never be removed from the files.
5. Other metadata should only be removed from files by agreement with their copyright holders.