A new version of Adobe Photoshop’s
The interpretation of unaltered sensor data from your camera involves at least the following steps:
- Demosaicing.
- White balance adjustment.
- Colorimetric interpretation.
- Gamma correction (also known as tone mapping).
- Noise reduction, antialising, sharpening.
Today we’ll look at two of them; demosaicing and gamma correction.
If you shoot in Jpeg, the above steps are all performed in-camera and additonal steps then convert the image to 8-bit, compress it to Jpeg form and save it to storage media. If you photograph in RAW, the data is simply written to media and left for post-processing off camera. RAW is an umbrella term for the proprietary formats (Canon does it one way, Nikon another etc.) that data is saved. Adobe Photoshop’s
Demosaicing
Almost all digital cameras have light sensitive detectors arranged in a Bayer array, where the individual elements of the array are staggered with respect to each other like this:
Demosaicing is the mathematical process of converting the recorded RGB (red, green and blue) values at these different locations on the camera’s sensor into an array of co-located RGB values. The locations where these values exist are called pixels.
There are twice as many green elements in the array as red and blue elements because the human eye is more sensitive to green light than both red and blue light. Each elements is typically capable of encoding its respective color channel in 12-bits (4096 levels), or more on some cameras.
Demosaicing is a pretty complicated and hence processor-intensive step, and so digital cameras take shortcuts to convert a RAW file into Jpeg (or Tiff), thus sacrificing image quality in relation to conducting this step on a personal computer using Photoshop’s Adobe Camera RAW or similar.
Gamma correction (tone mapping)
A digital camera perceives light in a linear fashion. Twice as much light produces twice as large a response on the sensor. This property is known as linear gamma.
The human eye perceives light in a logarithmic fashion. Twice as much light is perceived as brighter, but not twice as bright. Conversely, eyes are more sensitive to shadow detail than a camera sensor.
Gamma correction is necessary to redistribute tonal information to more closely correspond to the way the human eye perceives brightness. An example of such a gamma correction curve is shown below.
The above gamma curve remaps the camera brightness levels on the horizontal axis to different levels on the vertical axis in such a manner that shadow regions are stretched and higher brightness regions are compressed.
Click below to read more …
“There are twice as many green elements in the array as red and blue elements because the human eye is more sensitive to green light than both red and blue light.”
Maybe the P.I.A.S. would be a good place to reverse the course of perpetuating this ‘urban myth.’
Sensors in Sony Cameras, for example, do not have two green sensor filters in their array. The fourth one is something such as cyan. (I would prefer yellow–given Hering’s color theory, but I understand their reasoning.
‘…threw an old textbook at me, Human Color Vision by Robert Boynton. In it I found this explanation:
“To investigate ‘color,’” an experimenter puts a filter in front of a projector that is projecting an eye chart. “An observer, who formerly could read the 20/20 line, now finds that he or she can recognize only those letters corresponding to 20/60 acuity or worse. What can be legitimately concluded from this experiment? The answer is, nothing at all,” because the filter reduced the amount of light. “A control experiment is needed, where the same reduction in luminance is achieved using a neutral filter…. When such controls are used, it is typically found that varying spectral distribution has remarkably little effect upon visual acuity.”
In short, each cell in a Bayer sensor provides similar information about resolution. It is true that green light will provide a Bayer sensor with more information than red and blue light but that is only because the sensor has more green cells.’
Comment by Prof. Robert D. Jewell — February 19, 2006 @ 3:50 am
Robert,
I have to admit I don’t quite follow what you are saying.
We agree that most digital cameras have sensors arranged in a Bayer array?
That said, the sensors certainly aren’t colored as I state (for reason of simplification) … rather they are the same and there are colored filters which control what wavelengths of light are seen by particular elements of the array. There are typically twice as many of these green filters as red or blue ones and so twice as many elements of the array capture light in the green wavelength band.
Are we on the same page?
Comment by James Stephens — February 20, 2006 @ 10:23 am
Hello,
Just to update everyone .. in case anyone is interested … Bob and I are discussing this item offline via email.
Bob kindly clarified his point as follows:
“The question I want to raise is:
*Why* have twice as many greens?
The ‘urban myth’ is: ” because the human eye is more sensitive to green light than both red and blue light ”
The facts are that experimental attempts by psychology to confirm this claim, in fact, do not confirm it; but rather support its falsity. (This is the reason for the quotation from a classic work in psychology.)”
I replied back with
” Ah. Now I follow you .. and you raise an interesting point.
The suns energy spectrum peaks in the blue/green wavelengths and so it would make sense for our vision to be most sensitive to those wavelengths (and I would guess it is). So I could see having less red detectors than blue/green, Maybe Sony going with cyan makes sense.
Do you have any ideas yourself why it might be that there are twice as many green as red/blue? “
Bob has since been researching some texts but I just haven’t had the time to dive into this with my full attention as I am on vacation.
so hopefully in a couple of weeks ….
Over and out. Best,
James
Comment by James Stephens — February 23, 2006 @ 4:07 am
It’s surprising that the Bayer choices are not defined somewhere, but I can’t find any reference either. Someone decided on the array for, one presumes, good reason.
In the absence of an explanation I suggest that how TV handles colour may give some indications.
Image contours are found to be essentially defined by the eye’s response to luminosity (by the rods, in fact). Our resolution of spatial hue changes is much less fine. Colour TV therefore transmits a full bandwidth luminance signal, Y, accompanied by a pair of “colour diference signals”, U=0.493(B-Y) and V=0.877(R-Y), for the PAL standard, which give the deviation from grey but have less than a quarter of the bandwidth of the Y signal, i.e. less than half the resolution in both vertical and horizontal directions. The Y signal is the weighted sum of the RGB components from the camera, Y=0.3R+0.59G+0.11B, to match the eye’s sensitivity. (RGB components can obviously be regenerated from YUV for driving the three guns of the CRT.)
The similarity to the Bayer array is evident. The majority of cells would be expected to have a low saturation green, slightly yellow, filter, corresponding to Y and one would expect them to be interspersed with lower density arrays of cells having red and blue filters.
Now consider the gamma. I suggest that it’s another “urban myth” that the linear sensor response should be distorted to match the perceived response. Because the final image will also be perceived. It should therefore match the original exactly. So why is gamma correction applied?
Returning to TV or, more specifically to CRT displays, gamma correction is used, not to correct for human vision but to correct for the CRT’s square law response. As all current computers and operating systems assume CRT displays it follows that the same gamma correction will be required.
(It also follows that the printer driver will have to undo the correction, as it does.)
Comment by Michael Gay mieee — March 29, 2006 @ 2:53 pm
Michael,
Thankyou very much. I’m glad you brought in YUV to the discussion.
In the end it comes down to recording an image in such a way as to model human color perception as closely as possible. YUV was chosen for the PAL and NTSC broadcasting standards because it performs that task better than RGB for two main reasons:
1) It separates luminosity from color. Rods in the eye (present everywhere on the retina) are sensitive to luminosity.
2) Also, the luminosity, Y, is defined by ratios of red green and blue closely in proportion to the sensitivity of the eye to the band of wavelengths corresponding to those colors. The eye is most sensitive to green, followed by red and then blue
(see http://www.sns.ias.edu/~jns/images/spectral_response.jpg for a graphic of this response),
hence a ratio which favors green was chosen, namely Y=0.3R+0.59G+0.11B.
I’d never thought of it this way before but I guess a Bayer array is Y=0.25R+0.5G+0.25B. The urban myth is really true ……
On the gamma issue I don’t quite follow your reasoning. As far as I understand it, gamma correction on a camera’s sensor corrects the linear sensor response to that of the eye and gamma correction on a CRT display device corrects the displays square law response (to the gamma corrected sensor data) to that of the eye.
Best,
James
Comment by James Stephens — March 30, 2006 @ 3:25 pm
James,
After my last message I did the blindingly obvious, which, of course, I hadn’t thought of, and looked up the Bayer patent, US3971065. He’s quite clear, the majority green array is for luminance and the lower resolution red and blue for chrominance.
Concerning gamma, the point is that the gamma response is inherent to the CRT and must be corrected by the inverse law (gamma correction)to produce a displayed image which maps the original scene correctly. (LCD monitors have non-linear drive circuits to match their response to that of a CRT, for compatibility.)
Serenpenditiously our perception follows a similar law to the gamma correction, so in jpeg, where the data is reduced to 8 bits, interpolation with gamma correction may be advantageous.
A couple of references I found are
http://lcavwww.epfl.ch/~sabines/CI2001/L1.pdf
which gives the equations for gamma correction and human perception and
http://www.xbitlabs.com/articles/other/display/lcd-guide_13.html
which gives a comprehensive summary.
Best regards,
Michael Gay
Comment by Michael Gay mieee — March 31, 2006 @ 3:25 am
Thanks again Michael.
I’m with you on the luminosity thing.
In relation to gamma correction I’ll certainly take a look at those references when I get a chance (this weekend probably).
Best,
James
Comment by James Stephens — March 31, 2006 @ 8:29 am
Can some recommend a book on demosaicing, Auto white balance (AWB), color matrix correction, gamma correction etc… I am developing a camera for an instrument but the output is raw RGB and I don’t know how to interpolate the RGB colors to get the right pixel color. Theh camera resolution is 2MPix.
Can anyone either recommend a book, or tell me how to do the demosaicing interpolation? I would be very grateful.
Regards,
Robert
rjrodrig@yahoo.com
Comment by Rob — April 28, 2007 @ 1:39 pm