Multi-Space Color Correction: The New Paradigm

Our last technical blog post talked about color management, including considerations for maintaining color accuracy through the post process by keeping displays & projectors calibrated and understanding how each application manages the colors you see.  We alluded to the fact that at face value it can seem pretty complicated to manage color between all of the different kinds of camera technologies and technical standards, especially when dealing with multiple delivery specifications.

With this post we want to go into more detail about the issue of color grading for multiple standards, and how the new paradigm for color correction simplifies the process and keeps your grades more futureproof.  We’re going to examine a few things today.  First, how do computers process colors; second, where the problems we’re trying to fix come from; third, what the solutions are; and lastly, how to implement the solutions and why.


Color Engines

In previous posts (here, here, and here) we’ve mentioned the importance of using a color space agnostic color engine when doing specific color correction tasks - something like DaVinci Resolve Studio.  As a quick review, a color space agnostic engine is like a glorified R-G-B or R-G-B-Y calculator: It takes a specific set of decoded R-G-B or R-G-B-Y data, applies a specific set of mathematical operations based on the user input, and outputs the new R-G-B or R-G-B-Y data.

Agnostic color engines don’t care about what the data is supposed to mean, instead simply producing the results of an operation.  It’s up to the user to know if the results are right, or if the operation has put values out of spec or create unwanted distortions.  This is a double edged tool, since it places far more importance on user understanding to get things right, while being powerful enough to apply its corrections to any combination of custom situations.

As an example of how an agnostic engine works, let’s look at three of the simplest color correction operations: lift, gamma, and gain, operating strictly on the brightness (Y) component of the image.

Lift operates essentially as a global addition value: add or subtract a specific amount to each pixel’s value.  Because of the way that traditional EOTFs work and our human perception of brightness changes, lift tends to have the greatest effect on the darks: quickly raising or lowering the blacks while having a much more reduced effect on the mids and lights.

Gain operates essentially as a global multiplication value: multiply or divide the value of each pixel by a specific amount.  Since the operation essentially affects all tones within the image evenly, all parts of the image see a similar increase or decrease in brightness, though once again because of the EOTF considerations it has the greatest effect in the brights.

Gamma operates as an exponential value adjustment, affecting the linearity of values between the brights and the darks.  Lowering the gamma value has the effect of pulling more of the middle values towards the darks, while raising the gamma value has the effect of pushing the middle values brighter.  Once again, it still affects the brights and the darks, but at a much lower rate.

Notice that these operations don’t take into account what the data is supposed to mean.  And with new HDR EOTFs, especially with the Perceptual Quantization EOTF, you may find extreme changes across the image with very small values, which is why I recommend adding a roll-off curve as the last adjustment to your HDR grading workflow.

The combination of lift, gamma, and gain allow the colorist to adjust the overall brightness and contrast of the image, with fairly fine granularity and image control.

Compare these functions of an agnostic engine to their equivalents in a color space dependant engine.  In a color space dependent engine you’re more likely to find only two adjustments for controlling brightness and contrast: brightness and contrast.

The same color transformation operation has different effects on the image. Here, I've applied the same hue adjustment curve is applied to eight different color spaces and the effects on the vec

The brightness and contrast controls tend to be far more color space dependent, since they’re designed to affect the image in a way that more evenly affects the brightness or contrast along the expected EOTFs.  For the end user, this works a far simpler and often faster approach for minor corrections, at the expense of power, precision, and adaptability.  Which hasn’t been too bad of a trade off, so long as all digital video data operated in the same color space.

But adding support for new color spaces and EOTFs to a brightness and contrast operation requires rewriting the rules for how each of the new color spaces behave as digital RGB values.  That takes time to get right, and is oftimes not done at all.  Meaning that color space dependant engines tend to adapt more slowly to the emerging standards, and there’s no clear path for how to implement the upgrades.

Every color engine, whether we’re talking about a computer application or a chip found in a camera or display, makes assumptions about how to interpret the operations it's instructed to do.  Where the engines lie on the scale from fully color managed to completely color agnostic defines how the operations work, and what effect the ‘same’ color transformation has on the image.

The overall point here is that the same color transformations applied to different color spaces have different effects on the end image.  A hue rotation will accomplish something completely different in Rec. 709 than it will in Rec. 2020; standard gain affects HDR curves in ways that are somewhat unpredictable when compared with SDR curves.  Color engines can either try to compensate for this, or simply assume the user knows what he or she is doing.  And the more assumptions any single operation within an engine makes about the data, the more pronounced the differences if it’s applied to another color space.  These seemingly small differences can create massive problems in today’s color correction and color management workflows.


Understanding the Problem

With that background in mind, let’s explore where these problems come from.

Here’s something that may come as a shock if you haven’t dived into color management before: every camera ‘sees’ the world differently.  We’re not just talking about the effects of color temperature of light or the effects of the lenses (though those are important to keep in mind), but we’re talking about the camera sensors themselves.  All things being equal, different makes and models of camera will ‘see’ the same scene with different RGB values at the sensor.  In inexpensive cameras you may even see variation between individual cameras of the same make and model.

This isn’t a problem, it’s just how cameras work.  Variations in manufacturing process, decisions about which microfilters, microlenses, and OLPF to use, and the design of the sensor circuitry all play a part in changing the raw values the sensor sees.  But to keep things consistent, these unique camera RGB color values are almost always conformed to an output color standard using the camera’s image processor (or by the computer’s RAW interpreter) before you see them.

In the past, all video cameras conformed to analog video color spaces: NTSC/SMPTE-C, PAL, etc., and their early digital successors conformed to the digital equivalent standards: first Rec. 601 and then Rec. 709.

When it comes to conforming camera primaries to standard primaries, manufacturers had two choices: apply the look effects before or after the conforming step.  If you apply color transformations before the conforming step, you often have more information available for the change.  But by conforming first to the common color space, color correction operations would behave the same way between different camera makes and models.  

Most camera manufacturers took a hybrid approach, applying some transformations like gain and white balance before the conforming step, and then applying look effects after the conforming step.  And everything was golden, until the advent of digital cinema cameras.

Digital cinema class cameras started edging out film as the medium of choice for high quality television and feature film production a decade ago, and now vastly outnumber the quantity of film-first productions.  And here’s where we run into trouble.  Because digital cinema uses a different color space than digital video: DCI-P3.  Oh, and recently the video broadcast standards shifted to a much wider color space, Rec. 2020, to shake off the limiting shackles of the cathode ray tube.

Color space selection suddenly became an important part of the camera selection and workflow process, one that few people talked about.  Right from the get go the highest end cameras started offering multiple spaces that you could shoot and conform to, one of which was usually camera RGB.  But changing the conforming space means that any corrections or effects added to the image after conforming behaved differently than they did before, and many user generated looks would be color space dependent.

To fix this, many (but not all) digital cinema camera manufactures moved the ‘look’ elements of their color processing to before the conforming step.  This way, regardless of which color space you, as the operator, chooses any looks you apply will have the same effect on the final image.

Which is fine in camera, and fine through post production.  Unless your color correcting platform doesn’t understand what the primary color values mean, or if it can’t directly transform the values into your working space.  Then you need to create and add additional conforming elements as correction layers, which can increase the computational complexity and reduce the overall image quality.

Oh, and if you start working with multiple cameras with different look settings available, you can get into trouble almost instantly, since there isn’t usually a simple way of conforming all of them to your working space if it’s not Rec. 709.

Oh, and you may have to deliver to all of the different color spaces: Rec. 2020 for 4K television broadcast, DCI-P3 for your digital cinema delivery, Rec. 709 for HD Blu-ray and traditional broadcast, and Rec. 601 for DVDs.  And for sanitiy’s sake, let’s add HDR.

And don’t break the bank.


The Solution

What if there was a way to make sure that a) all of your looks would move simply between cameras, regardless of make and manufacturer, and b) you could color grade once and deliver in all formats simply, without needing to manage multiple grades?

There is a way to do it: create a new RGB color space that encompasses all possible color values, and do all of your color corrections there.

Here’s the block diagram:

Camera RGB -> Very Wide Gamut Working Space (Log or Linear) -> Color Correction / Looks -> Tone Map to Standard Space

By mapping all of the camera sensor values into the same log or linear space, with very wide RGB color primaries1, you can make sure that you have access to all of the image data captured by every camera and that all operations will have the same effect on all images.

But what do I mean by a “very wide gamut RGB space”?  There are two types of gamuts I’m talking about here, both of which have advantages and disadvantages.

The first kind is a color space with virtual RGB primaries:  the RGB color primaries land outside of the CIE 1931 color colorimetry diagram.  Remember that CIE 1931 maps the combinations of various wavelengths of light and the perceivable colors they produce onto an X-Y coordinate plane, and that any color space requires at least three primary color vertices on this chart.  But since the chart is bigger than all mapped colors, you can put these values outside of the actual set of real colors.

By putting the values outside of the visible color ranges you’re defining ‘red’, ‘green’, and ‘blue’ values that simply don’t and can’t exist. But they end up being quite useful, because when you map your primaries this way you can define an RGB color space that includes up to all possible color values.  Yes, you could simply use CIE XYZ values to map all colors, but all of the math needed for color manipulations would need to be redefined and rebuilt from the ground up (and it always requires at least 16 bits precision).  But an RGB space with virtual primaries allows you to use standard RGB math, while maintaining as many colors as possible.

Comparison of eight very wide gamut color spaces, five with virtual primaries, three with real (or mostly real) primaries.

Examples of color spaces using virtual RGB primaries include the color space defined by the Academy of Motion Picture Arts & Sciences, ACES AP0, and many manufacturer specific spaces like Sony S-Gamut3 / S-Gamut3.cine, ARRI Log C Wide Gamut, Canon Cinema Gamut, and the new RedWideGamutRGB found in RED’s IPP2.

The catch with these virtual primaries is that many operations that you as a colorist may be accustomed to doing won’t behave exactly the same way.  The reason being is that the RGB balance as it relates to hues and saturations doesn’t quite apply the same way.  Without getting mired in the details, the effects of these operations are related to the relative shape of the triangle produced by the RGB color primaries, and the color space triangles using all virtual primaries tend to be more dissimilar with the traditional RGB color spaces than the RGB spaces themselves.

So instead some wide gamut formats use all real, or mostly all real primaries to somewhat match the shape (i.e. color correction feel) of working with the smaller color gamuts.  A couple of examples here are Rec. 2020 (called Wide Gamut on 4K televisions), Adobe Wide Gamut, and ACES AP1.  While not covering all possible color values, these spaces cover very large portions of the visible color gamut, making them very useful for color correction working spaces.

Whichever very wide color space you choose to work in is up to you and your needs.  If your company or workflow requires ACES, use ACES.   If you’re only using one type of camera, such as a RED Weapon or an ARRI Alexa, you may find it beneficial to work in as specific manufacturer’s RGB space.

For most of the work we do here at Mystery Box that’s destined for anything other than web, I typically conform everything to Rec. 2020 and do my coloring and mastering in that space.  There are a couple of reasons for this:

  1. As a defined color spaces it uses real, pure wavelength primaries. Meaning that so long as only 3 color primaries are used for image reproduction, it’s about as wide as we’ll ever go.

  2. It encompasses 100% of Rec. 709 / sRGB and 99.98% of DCI-P3 (only losing a tiny amount of the reds)

  3. It encompasses 99.9% of Pointer’s gamut, a gamut that maps all real-world reflectable colors (not perceivable colors, just those found in the real world) onto the CIE XYZ gamut - essentially every color producible through the subtractive primaries.

  4. While it behaves differently than DCI-P3 and Rec. 709, they all behave fairly similarly so the learning curve is low.

  5. It requires fewer tone mapping corrections for the final output.

Whether these reasons are convincing enough for you or not is up to you.  Personally, I don’t find the 0.02% of DCI-P3 it doesn’t cover to actually matter, nor do the set of greens and blue-greens it doesn’t store (and no three color system can produce). These differences are so small that only in the absolute best side-by-sides in a lab could you hope to see a difference.

Whatever you do choose to use as a working space, it’s worth investing the time to pick one and stick with it.  Since the grading transformations do behave differently in the different color spaces, it’s easiest to pick one and refine your technique there to get the best possible results.


Conversions Implementation

Looking at the generalized workflow block diagram you’ll want to consider how to implement the different conversion steps for your own productions, in order to maintain the highest quality image pipeline with the lowest time and resource costs.  So let’s go into the two main places that you need to make new choices in the pipeline, and how to plan for them.

Conforming Camera RGB to Very Wide Gamut Working Space

Moving from Camera RGB to a very wide gamut space is a slightly different process for each camera system, and can depend on whether you’re capturing RAW data or a compressed video image.

When you’re using RAW formats, you’ll manage this step in the color correction or DIT software, which is the prefered workflow when image quality is paramount.  If you’re recording to non-raw intermediate format, like ProRes, DNxHR, or H.264 or any other flavor of MPEG video, you’ll need to select camera settings that best match your target wide gamut space.

Most RAW formats ignore camera looks applied by the operator and store the color decisions as metadata, but most video formats don’t.  Once again, camera settings vary, so it’s important to look at your specific system and run tests to find out where looks are applied in your camera’s image pipeline, and whether you can add a separate look on the video outputs for on-set monitoring while capturing a flattened LOG or linear image.

If your camera can’t separate the looks applied to the video files and the video output, and you want to capture a flat image but need to see it normalized on set, loading LUTs into on-set monitors is the ideal choice for image monitoring.  The process of creating and applying monitoring LUTs varies with your workflow, but we often find ourselves using a two-step process that uses Lattice to generate color space conversion LUTs, which we bring into DaVinci Resolve Studio to add creative looks and generate the final monitoring LUT.

Some cameras or DIT applications export their look settings as CDLs, LUTs, or other metadata for you to use later in the grading process, which you can then apply in post as the starting point for the grading process.  Again, workflows vary.

Generally you’ll want to move directly from camera RGB into the working space to preserve as much sensor information as possible.  That implies that you need to decide what your working space will be before capture (ACES AP0 or Rec. 2020 are recommended for future broadest compatibility), though it’s sometimes not an option.  While RAWs maintain camera primaries and allow you to jump directly into a wide working space later, if you’re forced to conform to standardized RGB for video intermediates you'll need to make that decision as early as possible.  In that case, put them into the widest color space available by the camera, whether that’s Rec. 2020, DCI-P3, or the manufacturer’s proprietary Wide Gamut space.

If RAWs aren’t an option, using a 12 bit log format video is your next best choice.  10 bit is fine too, but you won’t get as clean of corrections later, and may see some banding in fine gradients.  Anything less than 10 bits per channel create severe problems when color grading and really should only be used as a last resort.  When recording using an 8 bit format, you should only use a standard SDR EOTF (never LOG) - LOG with only 8 bits of precision can create MASSIVE amounts of banding.

To summarize: To maintain the highest image quality with the smallest resource pain, use RAW formats when possible, convert to the working or widest color space if you have to record as video files, and use LUTs on display outputs to avoid baking camera looks into the video data.

Tone Mapping the Working Space to Output

Moving from a wide working space to a final deliverable space is generally relatively simple process: simply convert each color value from the working space to the color value equivalent of the target space, and discard any data that lands outside of the target range.  In most Rec. 2020 -> DCI-P3, or Rec. 2020 -> Rec. 709 conversions, this is completely fine.  You may find minor clipping in a few of the most saturated colors, but overall you shouldn’t see too many places where the color is so bad you can’t live with it.

Where you do run into problems is when you’ve graded using an HDR transfer function and are moving into SDR.  A straight translation here results in very, very large amounts of clipping.  I haven’t mentioned EOTFs much yet, simply because most color engines where you’ll be doing wide gamut work use linear internals, since that tends to offer the most dynamic range and manipulation potential.

However, displays rarely offer a linear EOTF and so you’ll have to be monitoring in some transfer function or another.  Display monitoring is another reason I typically grade in BT.2020 (and usually in HDR), since displays need to be set to a specific color space and EOTF.  Which means that if you’re using a very wide working space, you must apply a tone map to your monitoring output, regardless of whether you’re grading in HDR or SDR (especially when you’re working in linear light).

The first series we published here on the blog about HDR video included a section on “Grading Mastering, and Delivering HDR”, where we presented a few bezier curves you can apply as the last element in your node structure for HDR grading in PQ or HLG.  These bezier curves are essentially luminance tone maps, converting the linear light values into the specific range of digital values you use for HDR.

A full tone map typically includes considerations for converting color information as well.  Just like the bezier curves control the roll off of the lights into your target range, tone maps roll off color values between color spaces, to minimize the amount of hard clipping.  Here its important to exercise caution and experiment with your specific needs before selecting a tone map, since this step can create hue or saturation shifts you don’t expect.

Tone mapping is the golden goose of simplifying multi-space color corrections.  It’s what brings everything together by making it possible to very, very quickly move from your working space into your delivery space.

If you grade in ACES AP0 or AP1, the tone maps are already prepared for your conversions.  Simply apply the tone map for the target system and voila, the conversion’s ready for rendering, preserving all (or rather most - they aren’t quite perfect) of the authorial intent of the grade.  We did this on our Yellowstone video to generate the HDR master.2

Grading in other wide color spaces often requires custom tone maps, or on-the-fly maps generated by a program such as DaVinci Resolve Studio.  RED Digital Cinema, for instance, has produced LUT based tone maps for converting their RedWideGamutRGB Log3G10 footage into various HDR and SDR color spaces.  The entire Dolby Vision format is essentially a shot by shot set of tone maps for various screen brightnesses.

Or, you may find yourself doing what we’ve done - spend time to create your own tone mapped LUTs for converting HDR and SDR of various formats, and refining these maps for each individual piece of HDR content so that you end up with the optimal SDR look for that work.


Why Bother?

Wide color space corrections and tone mapping for various output systems is the way that color correction will be treated and handled in the future.  With the arrival of BT. 2020 and HDR transforms, in just the last few years the number of delivery color encoding formats has increased three fold at the very least.  The only way to ensure your content will be compatible in the future is to adopt the new paradigm and multi-space coloring workflow.

DaVinci Resolve Studio’s latest update (release of version 14) saw a significant overhaul of the color management engine in the last few beta versions to optimize the core functionality for this kind of color management workflow.  If you’re using DaVinci Color Management or ACES color management in the latest version, DaVinci will automatically select the optimal RAW interpretation of your footage and conform it to your working space, removing the ambiguity of how to interpret your footage and maintaining the maximum image quality.

Another manufacturer who’s natively implemented a similar color pipeline is RED, with their new IPP2 color workflow.  They’ve moved all of their in-camera looks and apply them to the image data after the sensor RGB is converted into RedWideGamutRGB, tone mapping all outputs to your monitoring space.  With that they now allow you to select whether color adjustments in camera are burned into the ProRes files, or simply attached as a LUT or CDL.  This way, regardless of what your monitoring or eventual mastering space is, the color changes you make in camera will have the exact same effect across the board.

This is the workflow of the future.  Like with HDR, which we can assume will be the EOTF of the future, the efficiencies and simplicities of this particular workflow are so great that the sooner you get on board with it the better your position will be in a few years time.  Grading in ACES AP0 offers a level of future proofing that not even BT.2020 provides.  While BT.2020 still exceeds what current technologies can really do, ACES AP0 ensures that regardless of where color science heads in the future (4+ color primaries?), your footage will already be common format that’s simple to convert to the new standard, preserving all color data.

While there is a learning curve to this workflow, at a technical level it’s simpler to learn and apply than even understanding how HDR video works.  Yes, it takes some getting used to, but it’s worth learning.  Because in the end, you’ll find better quality than you can otherwise hope for.

Written by Samuel Bilodeau, Head of Technology and Post Production
 

Footnotes:

1 Yes, I’m making up the term “Very Wide Gamut” or “Very Wide Gamut RGB” simply because “Wide Gamut” and “Wide Gamut RGB” can refer to many different specific spaces, depending on the circumstances. Here I’m referring to any of these typical wide gamut spaces, or any space that covers a very large portion of the perceivable gamut.

2 A caveate note about ACES tone mapping: We used ACES AP0 with an ACEScc EOTF for our Yellowstone video. The tone mapping into HDR was fantastic and allowed me to skip my own range limiting map, and the ability to select different input transforms for each shot was fantastic. However, ACES failed when trying to generate an SDR version of the film: instead of tone mapping the higher dynamic range into the smaller SDR range, it clipped at the limits of SDR. This limitation makes me hesitant about recommending ACES for mixed-dynamic range work. It works wonderful for one or the other, but don't expect it to tone map directly between the two.

Display Calibration & Color Management

There are many different ways for consumers to experience your content today - so many that it’s often difficult to predict exactly where and how it’ll be seen.  Is it going to a theater?  Will it be watched on a television?  Which of the many handheld devices or personal computers will an end consumer use to view and listen to your work?  And how is that device or environment going to be rendering the colors?

Color management is an important consideration for every modern digital content production company to keep in the forefront of their minds.  In larger post production environments, there will often be a dedicated team that manages the preservation of color accuracy across the many screens and displays found throughout the facility.  But for small companies and independent producers, the burden of color management often falls on an individual with multiple roles, and is easier to ignore and to hope for the best than to spend the time and money to make sure it’s done right.

Before going any further, it’s important to define what we’re talking about when we say ‘color management.’  Color management is different than color correction or color grading, which is the process of normalizing colors and contrasts, maintaining color consistency across edits, and applying creative looks to your footage.  Instead, color management is about making sure the colors you see on your screens match as closely to the what the digital values stored in your video files are actually describing, within the color space you’re using.

In practice this means making sure that your displays, televisions, projectors, or other screens, as well as your lighting environment, are all calibrated so that their RGB balance, brightness, and contrast all match as close to the target standard as you can get them.  This makes sure that you don’t accidently add corrections to your digital data when you’re trying to ‘fix’ what you see on your displays that’s only there because of your displays or environment.  “Burning in” these kinds of shifts adversely affects the quality of your content by creating perceptual color shifts for your clients and consumers.

While calibration is essential, color management also involves ensuring the preservation of color from camera to end user display, keeping the color consistent between programs and ensuring your final deliverables contain the appropriate metadata.  Both parts to color management are essential, so we’re going to talk about both.  We’ll focus more on the calibration aspect of color management since that’s essential to get right, before briefly addressing color management in applications without getting mired too deep in advanced technical talk.


The problem

How do I know that my red is the same as your red?

This is one of the fundamental philosophical questions of color perception.  How do I know know that the way that I perceive red is the same as the way that you perceive red, and not how you perceive blue or green?  There’s actually no way to measure or determine for certain that the perceived shades are identical in the minds of any two individuals, since color perception happens as the brain interprets the stimulus it receives from the eyes.

While being a fun (or maddening) thought provoking question, color sameness is actually a really important baseline to establish in science and imaging.  In this case we’re not asking about the perception of color, but whether the actual shade of color produced or recorded by two devices is the same.  Today we’re only going to focus on colors being produced, and not recorded - we’ll cover capturing colors accurately in our next post.

There are a LOT of different kinds of displays in the world - from the ones we find on our mobile devices, to computer displays, televisions, and consumer or professional projectors.  The core technologies used to create or display images, such as plasma, LCD, OLEDs, etc., all render shades of color in slightly different ways, leading to differences in how colors within images look between displays.

But it’s not just the core technology used that affects the color rendition. Other factors like the age of the display, the specific implementation of the core technology (like edge-lit or backlit LCDs), the manufacturing tolerances for the specific class of display, the viewing angle, and the ambient environment all affect the colors produced or the colors perceived.  Which makes it almost impossible to predict the accuracy of color perception and rendering for one viewer, let alone the thousands or millions who are going to see your work.

But rather than throw up your hands in despair at the impossibility of the task, shift your focus to what you, as the content creator, can do: if you can be reasonably sure that what you see in your facility is as close to what’s actually being encoded as possible, you can be confident that your end viewers will not be seeing something horrifying.  While every end viewer’s experience will be different, at very least your content will be consistent for them - it will shift in exactly the same way as everyone else’s content, a shift they’re already used to and don’t even know it.

For that reason it’s important that when you master your work you’re viewing it in an environment and with a display that’s as close to perfectly accurate as possible.  But unfortunately, color calibration isn’t something you can simply ‘set and forget’: it needs to be done on a recurring schedule, especially with inexpensive displays.


What is Color Calibration?

How do we make sure color looks or is measured the same everywhere?

This question was first ‘answered’ in 1931 with the creation of the CIE XYZ color space.  Based the results of a series of tests that measured the sensitivity of the human vision to various colors, the CIE created a reference chart that mapped how the brain perceived the combination of visible wavelengths as colors into a Cartesian plane (X-Y graph).  This is called the CIE 1931 Chromaticity Diagram 

Three different color spaces referenced on the CIE 1931 Chromasticity diagram. The colors within each triangle represent the colors that can be produced by those three color primaries. All three share the same white point (D65).

This chart allows color scientists to assign a number value to all perceivable colors, both those that exist as a pure wavelength of light, and those that exist as a combination of wavelengths.  Every color you can see has a set of CIE 1931 coordinates to define its chromaticity (combined hue & saturation, ignoring brightness), which means that while we may not have an answer to a philosophical question of individual color experience, we do have a way of scientifically determining that my red is the same as your red.

This standard reference for colors is a powerful tool, and we can use it to define color spaces. A color space is the formal name for all of the colors a device can capture or produce using a limited set of primary colors.  Map the primary colors onto the chromaticity diagram, join them as geometric shape, and your device you can create or capture any color within the enclosed shape.  With an accompanying white point, you have the fundamentals ingredients for a defined color space, like Rec. 709, sRGB, AdobeRGB, etc.

Defining and adhering to color spaces is actually quite important to managing and matching end to end color.  Digital RGB values have no meaning without knowing which of the many possible shades of red, green, or blue color primaries are actually being used.  Interpreting digital data using different RGB primaries than the original creator used almost always results in nonlinear hue shifts throughout the image.

This is where color calibration comes in.  Color calibration is the process whereby a technician reads the actual color values produced by a display, and either adjusts the display’s settings to conform more closely to the target color space, and / or adjusts the signal coming to the display to better match the targeted output values.

To do this, you need access to four things:

  1. A signal generator to send the display specific digital values

  2. A colorimeter to measure the actual colors produced

  3. Control of the display’s input signal or color balance settings to adjust the output

  4. Software to manage the whole process and correlate the signal to measurement

If you want to make sure you’re doing it right, though, an in-depth understanding of how color and every image generation technology works helps a lot too.

Some consumer, most prosumer, and almost all professional displays leave the factory calibrated, though consumer and commercial televisions and almost all projectors must be calibrated after installation, for reasons we’ll talk about later.  Unfortunately, displays lose their calibration with time, and each kind and quality of display will start showing more or less variance as they age.  Which means that in circumstances where calibration is important, such as in professional video applications, displays require regular recalibration.

For desktop displays, this usually involves creating or update the ICC color profile, while for reference displays this typically involves adjusting the color balance controls so that the display itself better matches the target color space.

The differences in calibration technique comes from the workflow paradigm.  For desktop displays it’s assumed that the host computer will be directly attached to any number of different kinds of displays, each with their own color characteristics, at any given time - but always directly attached.  So, to simplify the end user experience, the operating system handles color management of attached displays through ICC profiles.

ICC profiles are data files that define how a display produces colors.  It records the CIE XYZ values of its RGB color primaries, white point, and black point, and its RGB tone curves, among some other metadata.

Using this information, the operating system “shapes” the digital signal sent to the display, converting on the fly the RGB values from the color space embedded in an image or video file into the display’s RGB space.  It does this for all applications, and essentially under all circumstances.  Some professional programs do bypass the internal color management, sort of, by assigning all images they decode or create to use the generic RGB profile (i.e. an undefined RGB color space). But it’s usually best to assume that for all displays directly attached to the computer, the operating system is applying some form of color management to what you’re seeing1.

Calibrating direct attached displays is relatively quick and easy.  The signal generator bypasses the operating system’s internal color management and produces a sequence of colored patches, which the colorimeter reads to map the display’s color output.  The software then generates an ICC color profile for that specific display, which compensates for color shifting from wear and tear, or the individual factory variances the display has.

Once calibrated, you can be reasonably confident that when viewing content, you’ll be seeing the content as close to intended as that particular display allows.

Reference displays, projectors, and televisions have a slightly different paradigm for calibration.  For calibrating computer displays, you can shape the signal to match the display characteristics.  But because of the assumption that a single video signal will (or at very least can) go to multiple displays or signal analysis hardware at the same time, and the signal generator is likely to have no information about the attached devices, it’s simply not practical to adjust the output signal.  Rather, professional output hardware always transmit their signals as pure RGB or YCbCr values, without worrying about the details of color space or managing color at all.

So instead of calibrating the signal, calibration of reference displays, projectors, or any kind of television usually requires adjusting the device itself.2

Once again, a signal generator creates specific color patches the colorimeter reads to see exactly what values the display creates.  Software then calculates the color’s offset as a Delta E value (how far away is the color produced from where it’s supposed to be according to the selected standard) and reports to the operator how far away from calibration it is.

The operator then goes through a set of trial and error adjustments to the image to lower the Delta E values of all the colors to get the best image possible.  Tweak the ‘red’ gain and see how that affects the colors produced.  Touch the contrast and see its effect on the overall image gamma - and on all the other colors.  Measure, tweak, measure, tweak, measure, tweak… and repeat, until the hardware is as close to the target color space as possible.

Calibration results showing DeltaE values for greyscale and color points

Generally, Delta E values less than 5 are good, less than 3 are almost imperceptible, and under 2 is considered accurate.  Once the calibration is complete, you can be reasonably sure that what you’re seeing on a reference display, projector, or television is as close to the target color space as possible.  But does that even matter?


Regular Calibration

Medium priced computer displays and professional reference displays usually leave the factory with a default calibration that puts them as close to standard reference as the technology allows.  The same is not true of most televisions and projectors - they leave the factory uncalibrated, or are in an uncalibrated mode by default for a couple of reasons which we’re not going to get into.

But even with this initial factory calibration for the displays that have it, the longer a display’s been used the more likely it will be experience color shifts.  How quickly it loses calibration depends on the technology in use: some technologies can lose their calibration in as short as a month with daily use.

The reasons behind this shift over time can be lumped together as “wear and tear”.  The exact reasons for each different display technology losing its calibration are a little off topic, so I’m going to spare you the gory details of the exact mechanisms that cause the degradations.  However, the important things to know are:

  1. The backlight of LCDs and the main bulb in digital projectors change colors over time. This is a major problem with the xenon arc lamps found in projectors, and is a bigger problem for CCFL LCDs than for LED lit (white or RGB) LCDs, but even the LED spectrums shift with use.

  2. The phosphors inside of CRTs and plasma displays degrade with time and their colors change, as do the primary color filters on LCD displays though at a slower pace.

  3. Anything using liquid crystals (LCD displays and LC or LCoS projectors) can suffer from degradation of the liquid crystal, which affects color and brightness contrasts.

  4. The spectrum of light emitted by plasma cells change with age, so they don’t stay balanced for the same output levels.

Or in other words, all displays change colors over time.  Setting up a regular calibration schedule for every display that you look at your content on is an important part of color management.  You don’t want to move a project from your reference display to your desktop to find that suddenly the entire video appears to be pulling magenta, or invite a client to review your work in your conference room to find the picture washed out or color shifted.


Environment and Color Management

Up until now we’ve been talking about the color characteristics of your displays and projectors.  But just as important as your display calibration is the characteristics of your environment in general.  The brightness level and color of lights in the room affect perceptions of contrast and the colors within the image.

This is really easy to get wrong.  Because not only does the display need to be calibrated for the target color space, it should be calibrated within the target environment.  The technician handling the calibration will usually make a judgement call for changing display values like display brightness, gamma curve, or white point based on these environmental choices.  But they may also make other recommendations about the environment to improve the perception of color on the screen - what to do to other displays, lighting, windows etc., so that your perception of color will better match industry standards.

Generally speaking, reference environments should be kept dim (not pitch black), using tungsten balanced lighting that’s as close to full spectrum as possible.  Avoid daylight balanced bulbs, and install blackout curtains on any windows.  Where possible, keep lighting above and pointed away from the workstation screens - reflected light is better than direct lighting, since it reduces glare and is better for color perception.

The easiest way get proper lighting is to set up track lighting with dimmable bulbs (LED or tungsten based, colored between 2800K & 3200K), and point the pots slightly away from the workstation.  The dimmer ensures that you can bring the environment into specification for grading, but can then bring the lighting back up to normal ambient conditions for general work or for installing hardware etc.  If changing the overhead lighting isn’t an option, good alternatives are stick lights on the opposite side of the room, positioned at standing height.

Keep your reference display or projector as the brightest screen in the environment.  If you don’t, your brights will look washed out and gray since they’re dimmer than other light sources. It will also affect your overall perception of contrast: you’ll perceive the image as darker and having more contrast than expected, and are therefore more likely to push up the mids and dark and wash out the image as a whole.  Dimming the brightness of interface displays, scopes, phones or tablets, and any other screen within the room will make sure that you’re perceiving the image on your reference hardware as accurately as possible.

Depending on the number of interface displays and other other light sources in the room, you may need to further lower ambient lighting to keep contrast perception as accurate as possible.  In rare cases, such as in small rooms, this may include turning the lights off completely since the interface displays provide sufficient ambient lighting for the environment.

Calibrating your displays is essential, calibrating the environment is important.  Usually it’s pretty easy to tweak environmental calibration for better color perception, so long as you’re starting from a dark or otherwise light controlled environment.  And unlike display calibration it’s something you can do once and not need to tweak for years.


Application Color Management

Once you’ve calibrated all of your hardware and your environment, it’s easy to assume that your job is done, and you don’t have to worry about color management until the next time you book a calibration session.  Oh how I wish that were the case.

Different applications manage color in different ways, which means you may still see differences between applications with the same footage.  Sometimes applications get in fights with the operating system over who’s managing color and both end up applying transformations you’re not aware of.

Which means it’s important to understand exactly how each application touches color.  To do that, let’s briefly look at how four common applications manage color: Adobe Premiere, Final Cut Pro X, Adobe After Effects, and DaVinci Resolve.

Both Adobe Premiere and Final Cut Pro X actively manage the colors within the project.  Adobe Premiere gives you exactly no ways of changing the color interpretation of the input files, beyond the embedded metadata in HEVC and a few other formats (NOT Apple ProRes).  It conforms everything to Rec. 709 in your viewers and signal outputs, and there’s no way to override this.  The operating system then uses the display’s ICC profile to conform the output so that you can see it as close to Rec. 709 as possible.  Which is good, because it means that when you output the video file, what you see is what you get.

Adobe Premiere’s color engine processes colors in 8 bit.  You can turn 16 bit color processing in the output or in the sequence settings by flagging on “Maximum Bit Depth” and “Maximum Render Quality.”  This is really important for using high bit depth formats like Apple ProRes, which stores 10 or 12 bit image data, assuming you want to maintain high color fidelity with your output files.  If you’re outputting to 8 bit formats for delivery you may still benefit from keeping these flags on, however, depending on how in depth your color corrections and gradients are.

Basically, Adobe Premiere assumes you know nothing about color management, and that it should handle everything for you.  Not a terrible assumption, just something to be aware of when you start thinking about managing color yourself.

Like Adobe Premiere, Final Cut Pro X also handles all of the color management, but offers at least a small amount of control over input and output settings.  By default, it processes colors at a higher internal bit depth than Premiere, and in linear color which offers smoother gradients and generally gives better results.  You also get to assign a working color space to your library and your project (sequence), though your only options are Rec. 709 and Wide Color Gamut (Rec. 2020).

Each clip by default is interpreted as belonging to the color space identified in its metadata, and conformed to the output color space selected by the project (sequence).  If necessary, you can override the color space interpretation of each video clip by assigning it to either Rec. 601 (NTSC or PAL), Rec. 709, or Rec. 2020 (notably missing is DCI-P3 and HDR curves).  When using professional video outs, the signal’s data levels of the is managed by the selection of Rec. 709 or Rec. 2020, and FCP-X handles everything else.  Like Adobe Premiere, it works with the operating system to conform the video displayed in the interface to the attached monitor’s ICC profile.

Both Adobe Premiere and FCP-X work on a “what you see is what you get” philosophy.  If your interface display is calibrated and using the proper ICC profile, you shouldn’t have to touch anything, ever.  It just works.  But gods Adobe and Apple forbid you try to make it do something else.

On the other hand, Adobe After Effects and DaVinci Resolve have highly flexible, colorspace agnostic color engines that allow you to nearly completely ignore all color management.  They’re quite content to simply apply the transformations you’ve requested to the digital data read in, and to not care about what color space or contrast curve the digital data is in.  And when you output, it simply writes the RGB data back to a file and you’re good to go.

Of course, that’s the theory.  After Effects makes a few color assumptions under the hood about intent, including ignoring the display ICC profile on output, since it has no idea what color space you’re working in anyway.  That sounds innocuous, but it’s a problem if you’re using a display with properties that are mismatched to the color profile of the footage you’re using3.  Suddenly your output, with an embedded color profile and playing back in a color managed application, may look significantly different than it did in After Effects.

Turning on After Effect’s color management by assigning a project working space allows for a more accurate viewing of the final output.  You can then flag on the view option to “Use Display Color Management” (on by default), and adjust the input space of any RGB footage.  But you can still get into trouble: any chroma subsampled footage, like ProRes 422 or H.264, is only permitted to use the embedded color profile.  Also Adobe ignores ProRes metadata for Rec. 2020 and HDR, which will negatively affect the output when using color management.  It also exhibits strange behavior when using HDR gamma curves and in some other working spaces.

DaVinci Resolve has some of the best functionality for color management.  It’s agnostic color engine renders color transformations in 32 bit float precision, and outputs raw RGB data to your video out.  It assumes you know what color space you’re using, so it’s happy to ignore everything else.  By default, on a Mac it applies the monitor ICC profile to the interface viewers, with the assumption that your input footage is Rec. 7094.

Fortunately, changing the working space is incredibly easy, even without color management turned on - simply set it the color primaries and EOTF in the Color Management tab of the project settings.  With color management off, this will only affect the interface display viewers, and then only if the flag “Use Mac Display Color Profile for Viewers” is set (on by default, MacOS only).  Unfortunately it does not as of yet apply ICC profiles to the viewers under Windows (see footnote 4).

When you turn DaVinci Resolve’s color management on, you have extremely fine grained control over color space - being able to set the input, working, and output color spaces and gammas separately (with Resolve managing the transformations on the fly), and then being able to bypass or override the input color space and gamma on a clip by clip basis in the color correction workspace.  And because of their 32 bit floating point internals, their conversions work really well, preserving “out of range” data between nodes and between steps in the color management process, allowing the operator to reign it in and make adjustments to the image at later steps - an advantage of active color management over LUTs in a few cases.

Input Processing Output Display
Adobe Premiere Assumes embedded or Rec. 709, cannot be changed 8 bit Rec. 709 with Gamma 2.4 assumed, 16 bit and linear color processing possible Rec. 709 on all outputs Output conformed to display using ICC profile
Final Cut Pro X Assumes embedded or Rec. 709, overridable to Rec. 2020 10-12 bit Rec 709 or Rec 2020 (configured by library) with gamma 2.4. Rec. 709 or Rec. 2020 on all outputs (configured by project) Output conformed to display using ICC profile
Adobe After Effects Assumes embedded or Rec. 709, ignored by default, reassignable for RGB formats but fixed interpretation of YCbCr 8 or 16 bit integer or 32 bit float agnostic color engine. Working space assignable on project basis, many fixed working spaces available RGB output in working space or generic RGB Color space and calibration defined by display (Pro out), output conformed to display using ICC profile for direct attached interfaces when working space assigned.
DaVinci Resolve Studio Ignored by default, global assignable with per-clip overrides to nearly any color space 32 bit floating point agnostic color engine. Working space assignable on a project basis, many combinations of working spaces with independently assignable color primaries and EOTFs RGB output in working space or assignable output space, or generic RGB Color space and calibration defined by display (pro out), output conformed to display using ICC profile for direct attached interfaces when working space assigned, LUTs available for pro output calibration.

These four programs kind of form a good scale for understanding application color management.  Generally speaking, the easier an application is to set up and use, the more hands-off management it’s likely to do, and give you anywhere from no, to very limited control over color management.  More advanced programs usually offer more in depth color management features, or the ability to bypass color management completely so that you’re able to have the finesse you need.  They also tend to preserve RGB data internally (and output that RGB data through professional video output cards), but require more of a knowledge of color spaces and the use of calibrated devices.

Calibrating your displays is a significant portion of the color management battle, though it’s also necessary to understand exactly what the applications are doing to the color if you want to be able to trust that what you’re seeing on the screen is reasonably close to what will be delivered to a client or to the end user.


What A Fine Mess We’re In

Keeping displays and projectors calibrated and trusting their accuracy has always been a concern, but it’s really become a major issue as the lower cost of video technologies has made the equipment more accessible, and since both the video and film production industries have shifted into modern digital productions.

“Back in the day”, analog video displays relied on color emissive phosphors for their primary colors.  The ‘color primaries’ of NTSC and PAL (and SECAM) weren’t based on the X-Y coordinates on the CIE XYZ 1931 diagram, but on the specific phosphors used in the CRT displays that emitted red, green, and blue light.  They weren’t officially defined with respect to the CIE 1931 standards until Recommendation BT.709 for High Definition Television Systems (Rec. 709) in 1990.

Around that time, with the introduction of liquid crystal displays computer displays also had to start defining colors more accurately.  They adopted the sRGB color space in mid to late nineties, using the same primaries as Rec. 709 but with a different data range and more flexible gamma control.  Naturally, both of these standards based their color primaries on… the CRT phosphors used in NTSC and PAL televisions systems.  And while the phosphors degrade and shift over time, they don’t shift anywhere near as much as the backlights of an LCD.  Meaning that prior to the early 2000s, when LCDs really took off, calibration was far less of an issue.

Now we have to worry not only about the condition of the display and its shifting calibration, but which of the multiple color spaces and new EOTFs (gamma curves) the display or application works with, what client deliverables need to be, and which parts of the process may or may not be fully color managed with our target spaces supported.

And then we have film.  Right up until the advent of end to end digital production, film had the massive benefit of “what you see is what you get” - your color space was the color space of the film stock you were using for your source, intermediates, and masters.  Now with the DCI standard of using gamma corrected CIE X’Y’Z’ values in digital cinema masters, you have to be far more cautious of projector calibration: it’s not possible to convert from CIE X’Y’Z’ into proper color output without regularly measuring the projector’s actual output values.  And we’re not going to talk about the nightmare of DCI white points and desktop displays that use the DCI-P3 color space.

Oh, and by the way, every camera sees the colors differently than the actual color spaces you’re trying to shoot in, and may or may not be conforming the camera color primaries to Rec 709, DCI-P3, or something else.  Because this needed to be more complicated.

Fortunately, with a basic understanding of color management and color calibration navigating the modern color problems is actually much more manageable than it all appears on face value.  In our next post we’re going to be discussing RED Digital Cinema’s Image Processing Pipeline 2 (IPP2), and why it’s the perfect paradigm for solving the modern color management problem.


But in the meantime, if you’re working in the Utah area and want to figure out the best way of calibrating your workspace or home office, give us a call.  We’ve got the right equipment and know how to make sure that when you look at your display or projector, you’re seeing as close to the standards as possible.

Color and deliver with confidence: make sure it’s calibrated.
 

Written by Samuel Bilodeau, Head of Technology and Post Production


ADDENDUM:

Color management and calibration are trickier than I’ve made it sound.  I’ve simplified a few things and tried to be as clear as possible, but there are many, many gotcha’s in the process of preserving color that can make it maddening.  And this is one area where a small amount of knowledge and trying to do things yourself can get you into huge amounts of trouble really quickly.

Trial and error is important to learning, and often it’s still the only way to feel out exactly what an application is doing to your files.  But be smart: calibrate your displays and let the programs manage things for you, unless you’re intending on experimenting and know the risks associated with it.

 

Footnotes:

1 Note, this is not a bad thing.  In most cases it’s a good thing.  It’s just something to be aware of and to understand how it works.

2 It’s also possible to use lookup tables to shape the signal for viewing on a reference display.  Here, the software will measure the actual values produced by the display, and calculate the offsets as values to put in a 3D LUT.  When attached to multiple displays using the same professional signals, LUTs should be applied using external hardware, when attached to one display only it’s acceptable to apply the LUT in the software application generating the output signal or in a hardware converter.  Ensure that the LUT is not applied to any place on the signal upstream of the final output recording.

3 This is a big problem with the iMac, or any other Wide Gamut / DCI-P3 display.  Colors will look different than expected without enabling color management within After Effects.

4 At least it did, until DaVinci Resolve Beta 14b8, 14b9, and 14.0 release - the option to flag on and off color management for the display disappeared with this update and I haven’t had time to test whether it’s on by default, works under Windows, or whether they’ve gone a different way with their color management.

HDR Video Part 1: What is HDR Video?

It’s October 2016, and here at Mystery Box we’ve been working in HDR video for a little over a year.

While it’s easier today to find out information about the new standard than it was when I first started reading the research last year, it’s still not always clear what it is and how it works.  So, to kick off our new weekly blog here on mysterybox.us, we’ve decided to publish five posts back-to-back on the subject of HDR video.  This is Part 1: What is HDR Video?

HDR video is as much of a revolution and leap forward as the jump from analog standard definition, to digital 4K.

Or, to put it far less clinically, it’s mind-blowingly, awesomesauce, revolutionarily, incredible!  If it doesn’t get you excited, I’m not sure why you’re reading this…

So what is it about HDR video that makes it so special, so much better than what we’ve been doing?  That’s what we’re going to dive into here.


HDR Video vs. HDR Photography

If you’re a camera guy or a even an image guy, you’re probably familiar with HDR photography.  And if you’re thinking “okay, what’s the big deal, we’ve had HDR for years”, think again.  HDR video is completely unrelated to HDR photography, except for the ‘higher dynamic range’ part.

In general, any high dynamic range technique seeks to capture or display more levels of brightness within a scene, that is, increase the overall dynamic range.  It’s kind of a ‘duh’ statement, but let’s go with it.

In photography, this usually means using multiple exposures at different exposure values (EVs), and blending the results into a single final image.  The catch, of course, has always been that regardless of how many stops of light you capture with your camera or HDR technique, you’re still limited by the same 256 levels of brightness offered by 8 bit JPEG compression and computer/television displays, or the slightly bigger, but still limited set of tonality offered by inks for print.

So, most HDR photography relies on creating regions of local contrast throughout the image, blending in the different exposure levels to preserve the details in the darks and the lights:

Photograph with standard contrast vs. the same with local contrast

While the results are often beautiful, they are, at their core, unnatural or surreal.

HDR Video is Completely Different

Instead of trying to compress the natural dynamic range of a scene into a very limited dynamic range for display, HDR video expands the dynamic range of the display itself by increasing the average and peak display brightnesses (measured in nits), and by increasing the overall image bit depth from 8 bit to at least 10 bits / channel, or from 255 brightness levels & 16 million colors, to at least 1024 brightness levels & 1.02 billion colors.

Standard Video / Photography Range vs. HDR Photography vs. HDR Video Ranges

The change of the display light level allows for extended ranges of tonalities through the darks and the lights, so that the final displayed image itself is a more natural rendering of a scene, one that’s able to match the overall dynamic range of today’s digital cinema and film-sourced cameras. And perhaps more importantly, when fully implemented, HDR video will almost completely match the dynamic range of the human eye itself.

How big of a deal is it?  I can’t describe it better than my younger brother did the first time I showed him HDR video:

 

“I want to say that it’s like you’re looking through a window into another world, except that when you look through a window, it’s not as crisp, or as clean, or as clear as this”.

 

First Impressions to HDR Video

First Impressions to HDR Video


How did we get here?

So if HDR video is so much better than what we’ve been using so far, why haven’t we been using it all along?

And now, for a history lesson (it’s interesting; but it’s not essential to know, so skip down if you don’t care).

Cathode Ray Tubes as scientific apparatus and ‘display’ devices have been around in some form or another since the late 1880s, but the first CRT camera wasn’t invented until the late  1920s.  Early cameras were big with low resolutions; televisions were grainy, noisy, and low fidelity.

Things changed quickly in the early years of television. As more companies jumped on board the CRT television bandwagon, each created slightly different, and incompatible, television systems in an effort to avoid patent rights infringement.  These different systems, with different signal types, meant that home television sets had to match the cameras used by the broadcaster, i.e., they had to be the made by the same company.  As a result, the first broadcaster in an area created a local monopoly for the equipment manufacturer they sourced their first cameras from, and consumers had no choice.

Foreseeing a large problem when more people started buying televisions sets, and more broadcasters wanted to enter an area, the United States government stepped in and said that the diversity of systems wouldn’t fly - all television broadcasts and television sets had to be compatible.  To that end they created a new governing body, the National Television System Committee, or NTSC, which went on to define the first national television standard in 1941.

We’ve had to deal with the outcomes of standardization, good and bad, ever since.

The good, obviously, has been that we don’t have to buy a different television for every channel we want to watch, or every part of the country we want to live in (though transnationals are often still out of luck).  The bad is that every evolution of the standard since 1941 has required backwards compatibility: today’s digital broadcast standards, and computer display standards too, are still limited in part by what CRTs could do in the 1940s and 50s.

Don’t believe me?  Even ignoring the NTSC 1/1.001 frame rate modifier, there’s still a heavy influence: let’s look at the list:

  1. Color Space: The YIQ color space for NTSC and the YUV color space used in both PAL and SECAM are both based on the colors that can be produced by the short glow phosphors, which coat the inside of CRT screens and form the light and color producing element of the CRT.  In the transition to digital, YIQ and YUV formed the basis for Rec. 601 color space (SD Digital), which in turn is the basis for Rec. 709 (HD Digital) color space (Rec. 709 uses almost the same primaries as Rec. 601).

    And just in case your computer feels left out, the same color primaries are used in the sRGB display standard too, because all of these color spaces were display referenced, and they were all built on the same CRT technology.  Because up until the early 2000s, CRTs were THE way of displaying images electronically - LCDs were low contrast, plasma displays were expensive, and neither LEDs nor DLPs had come into their own.
     

  2. Transfer Function: The transfer function (also called the gamma curve) used in SD and HD is also based on the CRT’s natural light-to-electrical and electrical-to-light response.  The CRT camera captured images with a light-to-voltage response curve of approximately gamma 1/2.2, while the CRT display recreated images with a voltage-to-light response curve of approximately gamma 2.4.  Together, these values formed the standard approximate system gamma of 1.2, and form the basis for the current reference display gamma standard of 2.4, found in ITU-T Recommendation BT.1886.
     

  3. Brightness Limits: Lastly, and probably most frustratingly, color accurate CRT displays require limited brightness to maintain their color accuracy. Depending on the actual phosphors used for primaries, that max-brightness value typically lands in the 80-120 nits range.  And consumer CRT displays, while bigger, brighter, and less color accurate, still only land in the 200 nit max brightness levels.  For comparison, the brightness levels found on different outdoor surfaces during a sunny day land in the 5000-14,000 range (or more!).

    This large brightness disparity between reference and consumer display levels has been accentuated in recent years with the replacement of CRTs with LCD, Plasma and OLED displays, which can easily push 300-500 nits peak brightness.  Those brightness levels skew the overall look of images graded at reference, while being very intolerant of changes in ambient light conditions.  In short this means that with the current standards, consumers rarely have the opportunity to see content in their homes as filmmakers intended.

So, because of the legacy cathode ray tube, (a dead technology), we’re stuck with a set of legacy standards that limit how we can deliver images to consumers.  But because CRTs are a dead technology, we now have an opportunity where we can choose to either be shackled by the 1950s for the rest of time, or, to say “enough is enough,” and use something better.  Something forward thinking.  Something our current technology can’t even match 100% yet.  Something like, HDR video.


The HDR Way

At the moment, there two different categories and multiple standards covering HDR video, including CTA’s HDR 10 Media Profile, Dolby’s Dolby Vision, and the BBC’s Hybrid Log Gamma.  And naturally, they all do things just a little differently.  I’ll cover their differences in depth in Part 3: HDR Video Terms Explained, but for now I’m going to lump them all together and just focus on the common aspects of all HDR video, and what makes it different than video from the past.

There are four main things that are required to call something HDR video: ITU-T Recommendation BT.2020 or DCI-P3 color space, a high dynamic range transfer function, 10 bits per channel transmission and display values, and transmitted metadata.

BT.709, DCI-P3, and BT.2020 on CIE XYZ 1931

1. Color Space: For the most part, HDR video is seen by many as an extension of the existing BT.2020 UHD/FUHD and DCI specifications, and as such uses either the wider BT.2020 color gamut (BT.2020 is the 4K/8K replacement for BT.709/Rec.709 HD broadcast standards), or the more limited, but still wide, DCI-P3 gamut.

BT.2020 uses pure wavelength primaries, instead primary values based on the light emissions of CRT phosphors or any material.  The catch is, of course, we can’t fully show these in a desktop display (yet), and only the most recent laser projectors can cover the whole color range. But ultimately, the breadth of the color space covers as many of the visible colors as is possible with three real primaries*, and includes all color values already available in Rec.709/sRGB and DCI-P3, as well as 100% of Adobe RGB and most printer spaces available with today’s pigments and dyes.

2. Transfer Function: Where HDR video diverges from standard BT.2020 and DCI specs is in its light-level-to-digital-value and digital-value-to-light-level relationship, called the OETF and EOTF respectively.  I’m going to go into more depth on OETFs and EOTFs at another time, but for now what we need to know is that the current relationship between light levels and digital values is a legacy of the cathode ray tube days, and approximates gamma 2.4.  Under this system, full white digital value of 235 translates to a light output of between 80-120nits.

Extending this same curve into a higher dynamic range output proves problematic because of the non-linear response of the human eye: it would either cause severe stepping in the darks and lights, or it would require 14-16 bits per channel while wasting digital values in increments that can’t actually be seen.  And it still wouldn’t be backwards compatible, in which case, what’s the point?

So instead, HDR video uses one of two new transfer curves: the BBC’s Hybrid Log Gamma (HLG), standardized in ARIB STD-B67, which allows for output brightness levels from 0.01 nit up to around 5000 nits, and Dolby’s Perceptual Quantization (PQ) curve, standardized in SMPTE ST.2084, which allows for output brightness levels from 0.0001 nit up to 10,000 nits.

PQ is the result of direct research done by Dolby to measure the response of the human eye, and to create a curve where no value is wasted with no visible stepping between values.  The advantage of PQ is pretty clear, in terms of maximizing future output brightness (the best experimental single displays currently max out at 4000 nits; Dolby’s test apparatus ranged from 0.004 to 20,000 nits) and increasing the amount of detail captured in the darks.

HLG, on the other hand, provides a degree of backwards compatibility, matching the output levels of gamma 2.4 for the first 50% of the curve, and reserving the top 50% of the values to the higher light level output.  Generally, HLG content with a system gamma of 1.2 looks pretty close to standard dynamic range content, though it’s whites sometimes end up compressed and greyer than content mastered in SDR to begin with.

Footage graded in Rec. 709 and the same graded in HLG.

(Side note: I prefer grading in SMPTE ST.2084 because of the extended dynamic range through the blacks, and smoother roll-into the whites.)
 

3. Bit Depth: The new transfer curves accentuate a problem that’s been with video since the switch from analog to digital values: stepping.  As displays have gotten brighter, the difference between two code values (say, digital value of 25 and 26) is sometimes enough that we can see a clear distinguishing line between the two greys.  This is especially true when using a display whose maximum brightness is greater than reference standard, and is more common in the blacks than in the whites.

Both the BT.2020 and DCI standards already have requirements to decrease stepping by switching signal encoding and transmission from 8 bits per channel to 10 bits minimum (12 bits for DCI), allowing for at least a 4 times smoother gradient.  However, BT.2020 still permits 8 bit rendering at the display, which is what you’ll find on the vast majority of televisions and reference displays on the market today.

On the other hand, HDR video goes one step further and requires 10 bit rendering at the display panel itself; that is, each color sub pixel must be capable of between 876 and 1024 distinguishable light levels, in all operational brightness and contrast modes.

The reason that HDR requires a 10 bit panel while BT.2020 doesn’t, is that our eyes are more susceptible to stepping in the value of a color or gradient than to stepping in its hue or saturation: the eye can easily make up for lower color fidelity (8 bits per channel in BT.2020 space) by filling in the gaps, but with an HDR curve the jump in light levels between two codes in 8 bits per channel is big enough that it’s clearly noticeable.

Comparison between gradients step sizes at 8 bit, 10 bit, and 12 bit precisions (contrast emphasized)

4. Metadata: The last thing that HDR video requires that standard BT.2020 doesn’t, is metadata.  All forms of HDR video should include information about both the content and the mastering environment.  This includes which EOTF was used in the grade, the maximum and frame average brightnesses of the content and display, and which RGB primaries were used.  Dolby Vision even includes metadata to define, shot by shot, how to translate the HDR values into the SDR range!

Consumer display manufacturers use this information to adapt content for their screens in real time, knowing when to clip or compress the highlights and darks (based on the capability of the screen it’s being shown on), and for the automatic selection of operational mode (switching from Rec. 709 to BT.2020, and in and out of HDR mode, without the end user ever having to change a setting).

 

So, in summary, what does HDR video do differently?  Wider color gamuts, new transfer function curves to allow for a much larger range of brightnesses, 10 bits per channel minimum requirement at the display to minimize stepping, and the transmission of metadata to communicate information about the content and its mastering environment to the end user.

All of which are essential, none of which are completely backwards compatible.


Yes, but what does it look like?

Unfortunately, the only way to really show you what HDR looks like is to tell you to go to a trade show or post house with footage to show, or buy a TV with HDR capabilities and stream some actual HDR content.  Because when you show HDR content on a normal display, it does not look right:

Images in SMPTE ST.2084 HDR Video formats do not appear normal when directly brought into Rec. 709 or sRGB Gamma 2.4 systems

You can get a little bit of a feel for it if I cut the brightness levels of a standard dynamic range image by half, and put it side-by-side with one that more closely follows the HDR range of brightnesses:

Normalized & Scaled SMPTE ST.2084 HDR Video vs Rec. 709 with Brightness Scaled

But that doesn’t capture what HDR video actually does.  I don’t quite know how to describe it - it’s powerful, beautiful, clear, real, present and multidimensional.  There’s an actual physiological and psychological response to the image that you don’t get with standard dynamic range footage - not simply an emotional response to the quality of the image, but the higher brightness levels actually trigger things in your eyes and brain that let you literally see it differently than anything you’ve seen before.

And once you start using it on a regular basis, nothing else seems quite as satisfactory, no other image quite as beautiful.  You end up with a feeling that everything else is just a little bit inadequate.  That’s why HDR will very rapidly become the new normal of future video.


So that's it for Part 1: What is HDR Video?  In Part 2 of our series on HDR video, we’re going to cover what you need to grade in HDR, and how can you cheat a bit to get a feel for the format by emulating its response curve on your existing reference hardware.

Written by Samuel Bilodeau, Head of Technology and Post Production


Endnotes:

* While ACES does cover the entire visible color spectrum, it’s primary RGB values are imaginary, which means that while it can code for all possible colors, there’s no way of building a piece of technology that actually uses the ACES RGB values as its primary display colors.  Or in other words, if you were to try and display ACES full value RED, you couldn’t, because that color doesn’t exist.