Getting more colours with Bit Depth

Another area where there is a lot of misconception about colour is the topic of colour bit depth. Really there should be, because it is a simple the numbers of binary numbers (0 or 1) have to describe each primary colour (red, green, blue) in the pixel (colour channel) the more unique colours will be available (and the closer the steps between colours will be. It does not necessarily mean that a wider range of colours will be possible (ie colour gamut). This effect is most easily seen in the image histograms.

As an example and 8-bit coding system, of SRGB, gives each pixel up the 8 bits  or 2⁸ = 256 combination. By convention zero (0) is no colour (black for that channel) and 255 is the maximum intensity of colour in that channel. When all the three primary colours are combined there are 2^8*3  = 16,777,216 different colours definable for any given pixel. This is often described as “true colour”. This is often called the “bits per pixel” (bpp)to describe the sum of all three colour channels and that represents a pixel.

An interesting fact is most human eyes can only perceive around 10 million discreet colours, so displaying any image in more than 24 bpp will go unnoticed.

Most modern camera, will capture images in 8 bit SRGB (24 bpp) and standard .jpeg has this bit depth. Some higher end cameras now offer other colour spaces (Adobe RGB or ProPhoto) and greater bit depth. Remember you probably will not be able to see the difference in terms of enriched colour or image quality. The extra bit depth however can be very handy for post processing and particularly when “stretching” the tonal range of an underexposed RAW file (which can often lead to colour banding in the shadows). Much photo editing and computer graphics software can handle 16bit colour and  .tiff formats can be saved up to this bit depth.

The Cambridge in Colour site has a simple tutorial of Bit Depth, including a great visualization of the effect of bit depth. The African Shutha project has a wonder summary of the topic my Graeme Cookson.

Perfectly Clear why I am saving in .tiff

The problems on my HP Spectre (death by update) have forced me back onto my toshiba laptop as the first place I review my photos, cull the duds and perhaps do some preliminary evaluation of what might be worthwhile to further process. I had got used to using AfterShot pro, and in particularly had found the built in plug-in perfectly clear a wonderful little short cut to see what was possible by refining RAW files. To start I would have to try and reproduce what perfectly clear had done. Then I realized is I exported the perfectly clear changes to a 16 bit .tiff file I would have the perfectly clear image as a starting point and still access to the extra colour depth and dynamic range. Also on the HP Spectre I had got to using lightroom much more, particular as the platform to run other plug-ins like OnOne and Nik Software. Both of which work well with .tiff files. The option of running both Aftershot and then lightroom on the laptop was getting a tediously slow.

Back on the slower laptop I still like to look at potential good photos in perfect clear. A couple of weeks ago I received an offer with a Perfectly Clear newsletter to download (for free) the Prefect Exposure plug-ins tools (for lightroom and photoshop) so I jumped at the opportunity. The lightroom style plug in is still an extra step inside Lightroom but it is wonderful time saver for a lot of the manual exposure slider tweaking and it can deliver the .tiff file back into the lightroom library. Just Perfect for further lightroom, Nik or OnOne adjustments, staying with the .tiff format of course. I strenuously avoid fixed workflows (I think they are anti-creative) but I know starting with perfectly clear can avoid a lot of tedium, even if I don’t use all of the changes it recommends. I will have some knowledge of a better place to start

If you haven’t looked at the Perfectly Clear plug-ins yet they are probably going to be the best things you can add to lightroom.

DNG and the Month of Living Dangerously

Well i’m glad September is over, I can reset the dial of my beloved Pentax back to RAW/PEF not RAW/DNG. This story began several months ago when I installed the new version of OnOne 10.5, and suddenly I started having problems with OnOne Browse not display RAW files. Initially it wasn’t every RAW file but as I tried reinstalling the update and specifically once I had installed version 10.5.1 I was not able to view any .pef (the Pentax native RAW file) other than for a fleeting moment when the file was first viewed. Also .tif files and .cr2 first would intermittently disappear. This made trying to use OnOne software standalone tedious and frustrating. My continued request for support were just answered with trite rubbish about updating drivers and re-installing the software. I got frustrated and stopped using OnOne for a long time.

I got used to using Aftershot Pro mainly for my initial review and perhaps quick processing in the field. Then loading up lightroom back home and finishing images. For a few photos I wanted to do extra post processing I used the plug-ins linked to either OnOne Enhance, OnOne Effects or Nik’s Software.There is really nothing wrong with Aftershot Pro, in fact despite OnOne’s claims to have the fastest RAW rendering I suspect aftershot might be faster. However I quiet liked being able to flag just the images I wanted to further post process and load only those into lightroom or just finish them in the other OnOne modules, as I had been able to in version 10.0.1.  This was ok and seldom more than two steps. usinf aftershot, then lightroom and then a plg-in was three and meant having to generate an intermediate exchange file. Then I remember that my Pentax is one of the very few cameras that can capture DNG files in camera. I had previously looked at using that format but the files where substantially bigger so I couldn’t see any benefit. However if I avoided the intermediate copies I felt I could still be ahead in terms of spaced consumed so I bravely decided to only take DNG files during September.

Where did all the space go?

 

In september I created 1420 photo files. Most RAW plus jpeg pairs and most of them RAW DNG with a few .cr2 files. They took up a whopping 85 GB (excluding any .tiff post processing versions)
compared with 3710 files taking up only 40gb for my  month long candian month long trip earlier.

Exploding DNG  Grenades

The first hint of an issue had showed up before September but I assumed it was an isolated accident. Occasionally, and I still can not figure out why or which piece of software is to blame, some of the DNG files will suddenly double in size. Typically it is my better shot that I am interested in. Then sometimes they double in size again. as an example most DNG file from the camera are around 23 MB each, then when they first explode they will reach around 54MB then the second explosion will see them at 120mb plus. It is only when I try to view one of these exploded files that the havoc begins. Aftershot nicely reports it has stopped working, and trying to restart it report that it shut down unexpectedly and is set to read that directory again but it nicely reports it is stopping again, and so on. Lightroom seems to disappear for a while and eventually just doesn’t acknowledge that these bigger DNG files exist. OnOne browser however goes into suspended animation and my computer totally locks up. If I kill the process, restart the computer OnOne will refuse toit refuses to restart.

I eventually worked out if I renamed the last directory I had used before such crashes I could restart the misbehaving software. However if I tried to re-access that directory with OnOne browse, it was back to sleep, and crashes again. Lightroom just kept ignoring the big files after long contemplation and aftershot was ok unless I tried to look more closely at those files. Picasa would still view these files but took a long time to load them, perhaps understandably because they were large. These problems occurred on both my new HP Spectre laptop and my HP desktop, but less so on tests on an old Toshiba laptop.

I soon came to the conclusion that these exploded DNG files were not good, possibly corrupted some how. Which left me with a risk, if I tried to view these files before I had a backup I could loose the RAW version (since I take RAW and jpeg I would at least still have the jpeg version). Unfortunately I realised this AFTER I lost a couple of rather nice photos. I do nothave decent DNG version of these now.

As the month progressed this just became a bigger and more frequent problem and I am no closer to knowing why. I just have a mass of DGN files I am reluctant to even veiw.

Toxic Tiffs

When I am working with plug-ins like OnOne or Nik’s software I had adopted the idea of using .tiff files which give me more colour bit depth and preserve the illumination dynamic range compared with Jpeg and are way smaller than using the .png format (photoshop file format). Again occasionally to start with these .tiff files derived from DNG files began to refuse to load. I could work on the file to begin with, but once i quit the plug-in concerned the file would not open. Worst once this occurred within a session all other tiff files I tried to access would suffer the same fate. Exiting the software and starting a new session sometimes fixed the problem till I hit another toxic .tiff. the file sized does not seem to be altered in the case of the .tif files, so there is no easyway to figure out which fie will or won’t open. So opening any .tif file has a risk!

At this stage I also started to suspect a photo/dng targeting virus or malware, but I can not findany reports of such a beast and my virus and malware scans do not report any problems,

So I have stopped recording in DNG for four days now and guess what? no problems for two days!

 

Is the problem in the Pentax version of DNG, or is it in the software, or even combination of software I’m trying to use?

 

The fact that the new apple IPhone 7 will save optionally save in DNG format could see others finding similar problems. I’ll let the do the debugging. I’ve had enough for now no more DNG for me!

Popular Image formats

opular image formats, change with time. At the moment jpeg and raw are the dominant players in storage of photographs from digital cameras. Actually RAW is not a format per se it just means the camera manufacturers format for the complete exposure information has been stored directly from the camera sensor, it is general uncompressed (and therefore large), not enhanced in any way (it will usually look flat on first view) but will have a higher dynamic range and often a higher bit depth than the equivalent Jpeg image. RAW file are often described as digital negatives, which is not truly correct in detail but is a good analogy because, like a film negative a raw file typical require some development or processing before it can be shared on the web (usually after being exported as a jpeg file) or Printed.

Adobe’s DNG (Digital NeGative) is heavily promoted by Adobe as the “standard” RAW format, yet few camera manufactures, especially the bigger ones have not yet embraced it. I have a pentax which does offer DNG as an alternative RAW format, Leica and Hasselblad are the only others I know of that can save to DNG in the camera. DNG does have one significant advantage, in my view, the files can  be 15 to 30% smaller than the equivalent RAW file when it is compressed. So many photographers convert to this format as they upload there photos. Another advantage, which is a two edged sword, is that the metadata and details of how an image is processed can be save in the DNG file, where as no camera metadata and processing details normally have to be stored in a separate file (eg an .xmp side car file) for propriety RAW format. The hidden danger here is you can unintentionally overwrite your original image with new edit and potentially degrade your image in an unrecoverable way.This also makes the DNG file less like a negative and more like a digital print.

JPEG  (.jpg, or .jif) is by far the most popular format for the capture of images on cameras and smart phones and their display on the web. It uses lossy compression, which means some detail is discarded to enable smaller file sizes. The degree of compression is generally selectable and is a trade-off between size and image quality. There are several variation but the common format adopted by camera manufacturers embedded metadata about the camera setting in the file header. It also contains delails of the colour space to be used. Virtually all photographic software and browsers are be able to read and render jpeg files.

PNG was designed as a lossless successor to the gif format, largely because of l;licencing fears over the gif format, and it is probably still the most wisely used lossless format for web display being supported by all browsers. Unlike gif it does not support animation, but it does support extended pelleted colour scheme (eg 324bit RGB & 32bit RGBa). It does not store metedata about the image, which may explain why it have been ignored by camera manufactures Its popularity is mainly in computer graphics and illustration, but can be used for saving photographs to be displayed on the web.

The TIFF format is one of the first image formats, it can use lossless as well as lossy compression and does have higher bit depth for its colour rendition (ie 16bit rather than 8bit). It was for a while the format of choice for very large images and a few camera manufactures did have it available as an optional format. It can be processed by most (but not all) photo management software and is still popular in the graphics, digital illustration areas, as well as the output format of choice from some HDR programs. Most web browser will do a reasonable job rendering tiff files, The underlying tiff standard has nit had a major updated since the early 90’s, but it is another format under the control of adobe.

GIF was once a very popular format to display of images on the net, because of its generally smaller size. It is still popular for some aspects of web page design because it can incorporate transparent zones within the image. a variant of the format also allows animated graphics to be created, and these have become very popular (overly so). If you load 3 to 5 similar images into google+ it will create an autoawesome, which is an animated gif stop motion style animation from those photos. I’m not aware of any cameras that capture photos as gif (some wencams and scanner can). There has been a lot discussed recently on how to pronounce the name of this format I still favour Gif (the sound G sound as in giraffe) but apparently one of the developers of the form prefers Jiff (pronouced with a soft J as in Jill)

Webp is a new format, specifically developed for web display,that is being developed by google but as opensource. It offers significantly small files and faster download. It is available in both lossless and lossy formats to give greater compression removing some detail. It does provide for storage of metadata. It is not yet widely supported but the chrome browser and applications like everpic can display it now. And while on pronunciation this is often called "weppy"

The variation of RAW, DNG, GIF & TIFF are all propriety formats potentially under the control of specific companies, and some worry that if and when these companies fail or probably worse they suddenly enforce strict licensing on patents, then these formats "could" potentially become difficult to use and exchange. I suspect the large number of existing images in these formats will ensure ongoing support.

Finally this is not a comprehensive list of image format, see this wikipedia comparison page to find other formats, but the formats discussed here are probably the most common ones you will come across.

Show which formats do I use?

Thanks again to Jessica Hische for her great drop caps.

Working on Raw Colour

 

Whilst I haven’t blogged about RAW format lately. Its not that I haven’t been taking lots of RAW photos. Its more that I haven’t a really  found productive way to handle processing my RAW photos. I have a pentax camera and their default RAW format is PEF version of RAW is currently ignored by picasa at the upload phase. Thats ok I can read the card a second time in the Pentax Browser. However that inevitably involves a lot of tweaking and slider adjustments for each photo individually. Then I discovered if I copied these PEF files into a folder that picasa is monitoring the file is visualized, with a set of automatic “adjustments”, just what those adjustments are and how to turn them off is a mystery. The good news is picasa doesn’t modify the file at all. I Have noticed that in lower light the jpeg from the Camera and the Picasa interpreted RAW version are often quiet different. If the image showed promise I’d usually end up fiddling in the Pentax Browser and end up with something different again. When using RAW I usually focus most on getting realistic colours from the images greater colourspace depth.

I’d heard good things about RAW Therapee but only got around to downloading it today.There is a lot of good stuff to discuss. more for later posts, but what I immediately loved was its default process, such as white balance and brightness and that are easily modified and customized profiles easily stored. You see a preview and can still adjust it further is required. If you like the preview you can save it as a JPEG, TIFF & PNG File. If you use picasa you’ll have the original jpeg, the picasa version and the RAW therapee version all side by side (like above) so you can compare and export the version you want.