Display Calibration & Color Management

There are many different ways for consumers to experience your content today - so many that it’s often difficult to predict exactly where and how it’ll be seen.  Is it going to a theater?  Will it be watched on a television?  Which of the many handheld devices or personal computers will an end consumer use to view and listen to your work?  And how is that device or environment going to be rendering the colors?

Color management is an important consideration for every modern digital content production company to keep in the forefront of their minds.  In larger post production environments, there will often be a dedicated team that manages the preservation of color accuracy across the many screens and displays found throughout the facility.  But for small companies and independent producers, the burden of color management often falls on an individual with multiple roles, and is easier to ignore and to hope for the best than to spend the time and money to make sure it’s done right.

Before going any further, it’s important to define what we’re talking about when we say ‘color management.’  Color management is different than color correction or color grading, which is the process of normalizing colors and contrasts, maintaining color consistency across edits, and applying creative looks to your footage.  Instead, color management is about making sure the colors you see on your screens match as closely to the what the digital values stored in your video files are actually describing, within the color space you’re using.

In practice this means making sure that your displays, televisions, projectors, or other screens, as well as your lighting environment, are all calibrated so that their RGB balance, brightness, and contrast all match as close to the target standard as you can get them.  This makes sure that you don’t accidently add corrections to your digital data when you’re trying to ‘fix’ what you see on your displays that’s only there because of your displays or environment.  “Burning in” these kinds of shifts adversely affects the quality of your content by creating perceptual color shifts for your clients and consumers.

While calibration is essential, color management also involves ensuring the preservation of color from camera to end user display, keeping the color consistent between programs and ensuring your final deliverables contain the appropriate metadata.  Both parts to color management are essential, so we’re going to talk about both.  We’ll focus more on the calibration aspect of color management since that’s essential to get right, before briefly addressing color management in applications without getting mired too deep in advanced technical talk.


The problem

How do I know that my red is the same as your red?

This is one of the fundamental philosophical questions of color perception.  How do I know know that the way that I perceive red is the same as the way that you perceive red, and not how you perceive blue or green?  There’s actually no way to measure or determine for certain that the perceived shades are identical in the minds of any two individuals, since color perception happens as the brain interprets the stimulus it receives from the eyes.

While being a fun (or maddening) thought provoking question, color sameness is actually a really important baseline to establish in science and imaging.  In this case we’re not asking about the perception of color, but whether the actual shade of color produced or recorded by two devices is the same.  Today we’re only going to focus on colors being produced, and not recorded - we’ll cover capturing colors accurately in our next post.

There are a LOT of different kinds of displays in the world - from the ones we find on our mobile devices, to computer displays, televisions, and consumer or professional projectors.  The core technologies used to create or display images, such as plasma, LCD, OLEDs, etc., all render shades of color in slightly different ways, leading to differences in how colors within images look between displays.

But it’s not just the core technology used that affects the color rendition. Other factors like the age of the display, the specific implementation of the core technology (like edge-lit or backlit LCDs), the manufacturing tolerances for the specific class of display, the viewing angle, and the ambient environment all affect the colors produced or the colors perceived.  Which makes it almost impossible to predict the accuracy of color perception and rendering for one viewer, let alone the thousands or millions who are going to see your work.

But rather than throw up your hands in despair at the impossibility of the task, shift your focus to what you, as the content creator, can do: if you can be reasonably sure that what you see in your facility is as close to what’s actually being encoded as possible, you can be confident that your end viewers will not be seeing something horrifying.  While every end viewer’s experience will be different, at very least your content will be consistent for them - it will shift in exactly the same way as everyone else’s content, a shift they’re already used to and don’t even know it.

For that reason it’s important that when you master your work you’re viewing it in an environment and with a display that’s as close to perfectly accurate as possible.  But unfortunately, color calibration isn’t something you can simply ‘set and forget’: it needs to be done on a recurring schedule, especially with inexpensive displays.


What is Color Calibration?

How do we make sure color looks or is measured the same everywhere?

This question was first ‘answered’ in 1931 with the creation of the CIE XYZ color space.  Based the results of a series of tests that measured the sensitivity of the human vision to various colors, the CIE created a reference chart that mapped how the brain perceived the combination of visible wavelengths as colors into a Cartesian plane (X-Y graph).  This is called the CIE 1931 Chromaticity Diagram 

Three different color spaces referenced on the CIE 1931 Chromasticity diagram. The colors within each triangle represent the colors that can be produced by those three color primaries. All three share the same white point (D65).

This chart allows color scientists to assign a number value to all perceivable colors, both those that exist as a pure wavelength of light, and those that exist as a combination of wavelengths.  Every color you can see has a set of CIE 1931 coordinates to define its chromaticity (combined hue & saturation, ignoring brightness), which means that while we may not have an answer to a philosophical question of individual color experience, we do have a way of scientifically determining that my red is the same as your red.

This standard reference for colors is a powerful tool, and we can use it to define color spaces. A color space is the formal name for all of the colors a device can capture or produce using a limited set of primary colors.  Map the primary colors onto the chromaticity diagram, join them as geometric shape, and your device you can create or capture any color within the enclosed shape.  With an accompanying white point, you have the fundamentals ingredients for a defined color space, like Rec. 709, sRGB, AdobeRGB, etc.

Defining and adhering to color spaces is actually quite important to managing and matching end to end color.  Digital RGB values have no meaning without knowing which of the many possible shades of red, green, or blue color primaries are actually being used.  Interpreting digital data using different RGB primaries than the original creator used almost always results in nonlinear hue shifts throughout the image.

This is where color calibration comes in.  Color calibration is the process whereby a technician reads the actual color values produced by a display, and either adjusts the display’s settings to conform more closely to the target color space, and / or adjusts the signal coming to the display to better match the targeted output values.

To do this, you need access to four things:

  1. A signal generator to send the display specific digital values

  2. A colorimeter to measure the actual colors produced

  3. Control of the display’s input signal or color balance settings to adjust the output

  4. Software to manage the whole process and correlate the signal to measurement

If you want to make sure you’re doing it right, though, an in-depth understanding of how color and every image generation technology works helps a lot too.

Some consumer, most prosumer, and almost all professional displays leave the factory calibrated, though consumer and commercial televisions and almost all projectors must be calibrated after installation, for reasons we’ll talk about later.  Unfortunately, displays lose their calibration with time, and each kind and quality of display will start showing more or less variance as they age.  Which means that in circumstances where calibration is important, such as in professional video applications, displays require regular recalibration.

For desktop displays, this usually involves creating or update the ICC color profile, while for reference displays this typically involves adjusting the color balance controls so that the display itself better matches the target color space.

The differences in calibration technique comes from the workflow paradigm.  For desktop displays it’s assumed that the host computer will be directly attached to any number of different kinds of displays, each with their own color characteristics, at any given time - but always directly attached.  So, to simplify the end user experience, the operating system handles color management of attached displays through ICC profiles.

ICC profiles are data files that define how a display produces colors.  It records the CIE XYZ values of its RGB color primaries, white point, and black point, and its RGB tone curves, among some other metadata.

Using this information, the operating system “shapes” the digital signal sent to the display, converting on the fly the RGB values from the color space embedded in an image or video file into the display’s RGB space.  It does this for all applications, and essentially under all circumstances.  Some professional programs do bypass the internal color management, sort of, by assigning all images they decode or create to use the generic RGB profile (i.e. an undefined RGB color space). But it’s usually best to assume that for all displays directly attached to the computer, the operating system is applying some form of color management to what you’re seeing1.

Calibrating direct attached displays is relatively quick and easy.  The signal generator bypasses the operating system’s internal color management and produces a sequence of colored patches, which the colorimeter reads to map the display’s color output.  The software then generates an ICC color profile for that specific display, which compensates for color shifting from wear and tear, or the individual factory variances the display has.

Once calibrated, you can be reasonably confident that when viewing content, you’ll be seeing the content as close to intended as that particular display allows.

Reference displays, projectors, and televisions have a slightly different paradigm for calibration.  For calibrating computer displays, you can shape the signal to match the display characteristics.  But because of the assumption that a single video signal will (or at very least can) go to multiple displays or signal analysis hardware at the same time, and the signal generator is likely to have no information about the attached devices, it’s simply not practical to adjust the output signal.  Rather, professional output hardware always transmit their signals as pure RGB or YCbCr values, without worrying about the details of color space or managing color at all.

So instead of calibrating the signal, calibration of reference displays, projectors, or any kind of television usually requires adjusting the device itself.2

Once again, a signal generator creates specific color patches the colorimeter reads to see exactly what values the display creates.  Software then calculates the color’s offset as a Delta E value (how far away is the color produced from where it’s supposed to be according to the selected standard) and reports to the operator how far away from calibration it is.

The operator then goes through a set of trial and error adjustments to the image to lower the Delta E values of all the colors to get the best image possible.  Tweak the ‘red’ gain and see how that affects the colors produced.  Touch the contrast and see its effect on the overall image gamma - and on all the other colors.  Measure, tweak, measure, tweak, measure, tweak… and repeat, until the hardware is as close to the target color space as possible.

Calibration results showing DeltaE values for greyscale and color points

Generally, Delta E values less than 5 are good, less than 3 are almost imperceptible, and under 2 is considered accurate.  Once the calibration is complete, you can be reasonably sure that what you’re seeing on a reference display, projector, or television is as close to the target color space as possible.  But does that even matter?


Regular Calibration

Medium priced computer displays and professional reference displays usually leave the factory with a default calibration that puts them as close to standard reference as the technology allows.  The same is not true of most televisions and projectors - they leave the factory uncalibrated, or are in an uncalibrated mode by default for a couple of reasons which we’re not going to get into.

But even with this initial factory calibration for the displays that have it, the longer a display’s been used the more likely it will be experience color shifts.  How quickly it loses calibration depends on the technology in use: some technologies can lose their calibration in as short as a month with daily use.

The reasons behind this shift over time can be lumped together as “wear and tear”.  The exact reasons for each different display technology losing its calibration are a little off topic, so I’m going to spare you the gory details of the exact mechanisms that cause the degradations.  However, the important things to know are:

  1. The backlight of LCDs and the main bulb in digital projectors change colors over time. This is a major problem with the xenon arc lamps found in projectors, and is a bigger problem for CCFL LCDs than for LED lit (white or RGB) LCDs, but even the LED spectrums shift with use.

  2. The phosphors inside of CRTs and plasma displays degrade with time and their colors change, as do the primary color filters on LCD displays though at a slower pace.

  3. Anything using liquid crystals (LCD displays and LC or LCoS projectors) can suffer from degradation of the liquid crystal, which affects color and brightness contrasts.

  4. The spectrum of light emitted by plasma cells change with age, so they don’t stay balanced for the same output levels.

Or in other words, all displays change colors over time.  Setting up a regular calibration schedule for every display that you look at your content on is an important part of color management.  You don’t want to move a project from your reference display to your desktop to find that suddenly the entire video appears to be pulling magenta, or invite a client to review your work in your conference room to find the picture washed out or color shifted.


Environment and Color Management

Up until now we’ve been talking about the color characteristics of your displays and projectors.  But just as important as your display calibration is the characteristics of your environment in general.  The brightness level and color of lights in the room affect perceptions of contrast and the colors within the image.

This is really easy to get wrong.  Because not only does the display need to be calibrated for the target color space, it should be calibrated within the target environment.  The technician handling the calibration will usually make a judgement call for changing display values like display brightness, gamma curve, or white point based on these environmental choices.  But they may also make other recommendations about the environment to improve the perception of color on the screen - what to do to other displays, lighting, windows etc., so that your perception of color will better match industry standards.

Generally speaking, reference environments should be kept dim (not pitch black), using tungsten balanced lighting that’s as close to full spectrum as possible.  Avoid daylight balanced bulbs, and install blackout curtains on any windows.  Where possible, keep lighting above and pointed away from the workstation screens - reflected light is better than direct lighting, since it reduces glare and is better for color perception.

The easiest way get proper lighting is to set up track lighting with dimmable bulbs (LED or tungsten based, colored between 2800K & 3200K), and point the pots slightly away from the workstation.  The dimmer ensures that you can bring the environment into specification for grading, but can then bring the lighting back up to normal ambient conditions for general work or for installing hardware etc.  If changing the overhead lighting isn’t an option, good alternatives are stick lights on the opposite side of the room, positioned at standing height.

Keep your reference display or projector as the brightest screen in the environment.  If you don’t, your brights will look washed out and gray since they’re dimmer than other light sources. It will also affect your overall perception of contrast: you’ll perceive the image as darker and having more contrast than expected, and are therefore more likely to push up the mids and dark and wash out the image as a whole.  Dimming the brightness of interface displays, scopes, phones or tablets, and any other screen within the room will make sure that you’re perceiving the image on your reference hardware as accurately as possible.

Depending on the number of interface displays and other other light sources in the room, you may need to further lower ambient lighting to keep contrast perception as accurate as possible.  In rare cases, such as in small rooms, this may include turning the lights off completely since the interface displays provide sufficient ambient lighting for the environment.

Calibrating your displays is essential, calibrating the environment is important.  Usually it’s pretty easy to tweak environmental calibration for better color perception, so long as you’re starting from a dark or otherwise light controlled environment.  And unlike display calibration it’s something you can do once and not need to tweak for years.


Application Color Management

Once you’ve calibrated all of your hardware and your environment, it’s easy to assume that your job is done, and you don’t have to worry about color management until the next time you book a calibration session.  Oh how I wish that were the case.

Different applications manage color in different ways, which means you may still see differences between applications with the same footage.  Sometimes applications get in fights with the operating system over who’s managing color and both end up applying transformations you’re not aware of.

Which means it’s important to understand exactly how each application touches color.  To do that, let’s briefly look at how four common applications manage color: Adobe Premiere, Final Cut Pro X, Adobe After Effects, and DaVinci Resolve.

Both Adobe Premiere and Final Cut Pro X actively manage the colors within the project.  Adobe Premiere gives you exactly no ways of changing the color interpretation of the input files, beyond the embedded metadata in HEVC and a few other formats (NOT Apple ProRes).  It conforms everything to Rec. 709 in your viewers and signal outputs, and there’s no way to override this.  The operating system then uses the display’s ICC profile to conform the output so that you can see it as close to Rec. 709 as possible.  Which is good, because it means that when you output the video file, what you see is what you get.

Adobe Premiere’s color engine processes colors in 8 bit.  You can turn 16 bit color processing in the output or in the sequence settings by flagging on “Maximum Bit Depth” and “Maximum Render Quality.”  This is really important for using high bit depth formats like Apple ProRes, which stores 10 or 12 bit image data, assuming you want to maintain high color fidelity with your output files.  If you’re outputting to 8 bit formats for delivery you may still benefit from keeping these flags on, however, depending on how in depth your color corrections and gradients are.

Basically, Adobe Premiere assumes you know nothing about color management, and that it should handle everything for you.  Not a terrible assumption, just something to be aware of when you start thinking about managing color yourself.

Like Adobe Premiere, Final Cut Pro X also handles all of the color management, but offers at least a small amount of control over input and output settings.  By default, it processes colors at a higher internal bit depth than Premiere, and in linear color which offers smoother gradients and generally gives better results.  You also get to assign a working color space to your library and your project (sequence), though your only options are Rec. 709 and Wide Color Gamut (Rec. 2020).

Each clip by default is interpreted as belonging to the color space identified in its metadata, and conformed to the output color space selected by the project (sequence).  If necessary, you can override the color space interpretation of each video clip by assigning it to either Rec. 601 (NTSC or PAL), Rec. 709, or Rec. 2020 (notably missing is DCI-P3 and HDR curves).  When using professional video outs, the signal’s data levels of the is managed by the selection of Rec. 709 or Rec. 2020, and FCP-X handles everything else.  Like Adobe Premiere, it works with the operating system to conform the video displayed in the interface to the attached monitor’s ICC profile.

Both Adobe Premiere and FCP-X work on a “what you see is what you get” philosophy.  If your interface display is calibrated and using the proper ICC profile, you shouldn’t have to touch anything, ever.  It just works.  But gods Adobe and Apple forbid you try to make it do something else.

On the other hand, Adobe After Effects and DaVinci Resolve have highly flexible, colorspace agnostic color engines that allow you to nearly completely ignore all color management.  They’re quite content to simply apply the transformations you’ve requested to the digital data read in, and to not care about what color space or contrast curve the digital data is in.  And when you output, it simply writes the RGB data back to a file and you’re good to go.

Of course, that’s the theory.  After Effects makes a few color assumptions under the hood about intent, including ignoring the display ICC profile on output, since it has no idea what color space you’re working in anyway.  That sounds innocuous, but it’s a problem if you’re using a display with properties that are mismatched to the color profile of the footage you’re using3.  Suddenly your output, with an embedded color profile and playing back in a color managed application, may look significantly different than it did in After Effects.

Turning on After Effect’s color management by assigning a project working space allows for a more accurate viewing of the final output.  You can then flag on the view option to “Use Display Color Management” (on by default), and adjust the input space of any RGB footage.  But you can still get into trouble: any chroma subsampled footage, like ProRes 422 or H.264, is only permitted to use the embedded color profile.  Also Adobe ignores ProRes metadata for Rec. 2020 and HDR, which will negatively affect the output when using color management.  It also exhibits strange behavior when using HDR gamma curves and in some other working spaces.

DaVinci Resolve has some of the best functionality for color management.  It’s agnostic color engine renders color transformations in 32 bit float precision, and outputs raw RGB data to your video out.  It assumes you know what color space you’re using, so it’s happy to ignore everything else.  By default, on a Mac it applies the monitor ICC profile to the interface viewers, with the assumption that your input footage is Rec. 7094.

Fortunately, changing the working space is incredibly easy, even without color management turned on - simply set it the color primaries and EOTF in the Color Management tab of the project settings.  With color management off, this will only affect the interface display viewers, and then only if the flag “Use Mac Display Color Profile for Viewers” is set (on by default, MacOS only).  Unfortunately it does not as of yet apply ICC profiles to the viewers under Windows (see footnote 4).

When you turn DaVinci Resolve’s color management on, you have extremely fine grained control over color space - being able to set the input, working, and output color spaces and gammas separately (with Resolve managing the transformations on the fly), and then being able to bypass or override the input color space and gamma on a clip by clip basis in the color correction workspace.  And because of their 32 bit floating point internals, their conversions work really well, preserving “out of range” data between nodes and between steps in the color management process, allowing the operator to reign it in and make adjustments to the image at later steps - an advantage of active color management over LUTs in a few cases.

Input Processing Output Display
Adobe Premiere Assumes embedded or Rec. 709, cannot be changed 8 bit Rec. 709 with Gamma 2.4 assumed, 16 bit and linear color processing possible Rec. 709 on all outputs Output conformed to display using ICC profile
Final Cut Pro X Assumes embedded or Rec. 709, overridable to Rec. 2020 10-12 bit Rec 709 or Rec 2020 (configured by library) with gamma 2.4. Rec. 709 or Rec. 2020 on all outputs (configured by project) Output conformed to display using ICC profile
Adobe After Effects Assumes embedded or Rec. 709, ignored by default, reassignable for RGB formats but fixed interpretation of YCbCr 8 or 16 bit integer or 32 bit float agnostic color engine. Working space assignable on project basis, many fixed working spaces available RGB output in working space or generic RGB Color space and calibration defined by display (Pro out), output conformed to display using ICC profile for direct attached interfaces when working space assigned.
DaVinci Resolve Studio Ignored by default, global assignable with per-clip overrides to nearly any color space 32 bit floating point agnostic color engine. Working space assignable on a project basis, many combinations of working spaces with independently assignable color primaries and EOTFs RGB output in working space or assignable output space, or generic RGB Color space and calibration defined by display (pro out), output conformed to display using ICC profile for direct attached interfaces when working space assigned, LUTs available for pro output calibration.

These four programs kind of form a good scale for understanding application color management.  Generally speaking, the easier an application is to set up and use, the more hands-off management it’s likely to do, and give you anywhere from no, to very limited control over color management.  More advanced programs usually offer more in depth color management features, or the ability to bypass color management completely so that you’re able to have the finesse you need.  They also tend to preserve RGB data internally (and output that RGB data through professional video output cards), but require more of a knowledge of color spaces and the use of calibrated devices.

Calibrating your displays is a significant portion of the color management battle, though it’s also necessary to understand exactly what the applications are doing to the color if you want to be able to trust that what you’re seeing on the screen is reasonably close to what will be delivered to a client or to the end user.


What A Fine Mess We’re In

Keeping displays and projectors calibrated and trusting their accuracy has always been a concern, but it’s really become a major issue as the lower cost of video technologies has made the equipment more accessible, and since both the video and film production industries have shifted into modern digital productions.

“Back in the day”, analog video displays relied on color emissive phosphors for their primary colors.  The ‘color primaries’ of NTSC and PAL (and SECAM) weren’t based on the X-Y coordinates on the CIE XYZ 1931 diagram, but on the specific phosphors used in the CRT displays that emitted red, green, and blue light.  They weren’t officially defined with respect to the CIE 1931 standards until Recommendation BT.709 for High Definition Television Systems (Rec. 709) in 1990.

Around that time, with the introduction of liquid crystal displays computer displays also had to start defining colors more accurately.  They adopted the sRGB color space in mid to late nineties, using the same primaries as Rec. 709 but with a different data range and more flexible gamma control.  Naturally, both of these standards based their color primaries on… the CRT phosphors used in NTSC and PAL televisions systems.  And while the phosphors degrade and shift over time, they don’t shift anywhere near as much as the backlights of an LCD.  Meaning that prior to the early 2000s, when LCDs really took off, calibration was far less of an issue.

Now we have to worry not only about the condition of the display and its shifting calibration, but which of the multiple color spaces and new EOTFs (gamma curves) the display or application works with, what client deliverables need to be, and which parts of the process may or may not be fully color managed with our target spaces supported.

And then we have film.  Right up until the advent of end to end digital production, film had the massive benefit of “what you see is what you get” - your color space was the color space of the film stock you were using for your source, intermediates, and masters.  Now with the DCI standard of using gamma corrected CIE X’Y’Z’ values in digital cinema masters, you have to be far more cautious of projector calibration: it’s not possible to convert from CIE X’Y’Z’ into proper color output without regularly measuring the projector’s actual output values.  And we’re not going to talk about the nightmare of DCI white points and desktop displays that use the DCI-P3 color space.

Oh, and by the way, every camera sees the colors differently than the actual color spaces you’re trying to shoot in, and may or may not be conforming the camera color primaries to Rec 709, DCI-P3, or something else.  Because this needed to be more complicated.

Fortunately, with a basic understanding of color management and color calibration navigating the modern color problems is actually much more manageable than it all appears on face value.  In our next post we’re going to be discussing RED Digital Cinema’s Image Processing Pipeline 2 (IPP2), and why it’s the perfect paradigm for solving the modern color management problem.


But in the meantime, if you’re working in the Utah area and want to figure out the best way of calibrating your workspace or home office, give us a call.  We’ve got the right equipment and know how to make sure that when you look at your display or projector, you’re seeing as close to the standards as possible.

Color and deliver with confidence: make sure it’s calibrated.
 

Written by Samuel Bilodeau, Head of Technology and Post Production


ADDENDUM:

Color management and calibration are trickier than I’ve made it sound.  I’ve simplified a few things and tried to be as clear as possible, but there are many, many gotcha’s in the process of preserving color that can make it maddening.  And this is one area where a small amount of knowledge and trying to do things yourself can get you into huge amounts of trouble really quickly.

Trial and error is important to learning, and often it’s still the only way to feel out exactly what an application is doing to your files.  But be smart: calibrate your displays and let the programs manage things for you, unless you’re intending on experimenting and know the risks associated with it.

 

Footnotes:

1 Note, this is not a bad thing.  In most cases it’s a good thing.  It’s just something to be aware of and to understand how it works.

2 It’s also possible to use lookup tables to shape the signal for viewing on a reference display.  Here, the software will measure the actual values produced by the display, and calculate the offsets as values to put in a 3D LUT.  When attached to multiple displays using the same professional signals, LUTs should be applied using external hardware, when attached to one display only it’s acceptable to apply the LUT in the software application generating the output signal or in a hardware converter.  Ensure that the LUT is not applied to any place on the signal upstream of the final output recording.

3 This is a big problem with the iMac, or any other Wide Gamut / DCI-P3 display.  Colors will look different than expected without enabling color management within After Effects.

4 At least it did, until DaVinci Resolve Beta 14b8, 14b9, and 14.0 release - the option to flag on and off color management for the display disappeared with this update and I haven’t had time to test whether it’s on by default, works under Windows, or whether they’ve gone a different way with their color management.

HDR Video Part 1: What is HDR Video?

It’s October 2016, and here at Mystery Box we’ve been working in HDR video for a little over a year.

While it’s easier today to find out information about the new standard than it was when I first started reading the research last year, it’s still not always clear what it is and how it works.  So, to kick off our new weekly blog here on mysterybox.us, we’ve decided to publish five posts back-to-back on the subject of HDR video.  This is Part 1: What is HDR Video?

HDR video is as much of a revolution and leap forward as the jump from analog standard definition, to digital 4K.

Or, to put it far less clinically, it’s mind-blowingly, awesomesauce, revolutionarily, incredible!  If it doesn’t get you excited, I’m not sure why you’re reading this…

So what is it about HDR video that makes it so special, so much better than what we’ve been doing?  That’s what we’re going to dive into here.


HDR Video vs. HDR Photography

If you’re a camera guy or a even an image guy, you’re probably familiar with HDR photography.  And if you’re thinking “okay, what’s the big deal, we’ve had HDR for years”, think again.  HDR video is completely unrelated to HDR photography, except for the ‘higher dynamic range’ part.

In general, any high dynamic range technique seeks to capture or display more levels of brightness within a scene, that is, increase the overall dynamic range.  It’s kind of a ‘duh’ statement, but let’s go with it.

In photography, this usually means using multiple exposures at different exposure values (EVs), and blending the results into a single final image.  The catch, of course, has always been that regardless of how many stops of light you capture with your camera or HDR technique, you’re still limited by the same 256 levels of brightness offered by 8 bit JPEG compression and computer/television displays, or the slightly bigger, but still limited set of tonality offered by inks for print.

So, most HDR photography relies on creating regions of local contrast throughout the image, blending in the different exposure levels to preserve the details in the darks and the lights:

Photograph with standard contrast vs. the same with local contrast

While the results are often beautiful, they are, at their core, unnatural or surreal.

HDR Video is Completely Different

Instead of trying to compress the natural dynamic range of a scene into a very limited dynamic range for display, HDR video expands the dynamic range of the display itself by increasing the average and peak display brightnesses (measured in nits), and by increasing the overall image bit depth from 8 bit to at least 10 bits / channel, or from 255 brightness levels & 16 million colors, to at least 1024 brightness levels & 1.02 billion colors.

Standard Video / Photography Range vs. HDR Photography vs. HDR Video Ranges

The change of the display light level allows for extended ranges of tonalities through the darks and the lights, so that the final displayed image itself is a more natural rendering of a scene, one that’s able to match the overall dynamic range of today’s digital cinema and film-sourced cameras. And perhaps more importantly, when fully implemented, HDR video will almost completely match the dynamic range of the human eye itself.

How big of a deal is it?  I can’t describe it better than my younger brother did the first time I showed him HDR video:

 

“I want to say that it’s like you’re looking through a window into another world, except that when you look through a window, it’s not as crisp, or as clean, or as clear as this”.

 

First Impressions to HDR Video

First Impressions to HDR Video


How did we get here?

So if HDR video is so much better than what we’ve been using so far, why haven’t we been using it all along?

And now, for a history lesson (it’s interesting; but it’s not essential to know, so skip down if you don’t care).

Cathode Ray Tubes as scientific apparatus and ‘display’ devices have been around in some form or another since the late 1880s, but the first CRT camera wasn’t invented until the late  1920s.  Early cameras were big with low resolutions; televisions were grainy, noisy, and low fidelity.

Things changed quickly in the early years of television. As more companies jumped on board the CRT television bandwagon, each created slightly different, and incompatible, television systems in an effort to avoid patent rights infringement.  These different systems, with different signal types, meant that home television sets had to match the cameras used by the broadcaster, i.e., they had to be the made by the same company.  As a result, the first broadcaster in an area created a local monopoly for the equipment manufacturer they sourced their first cameras from, and consumers had no choice.

Foreseeing a large problem when more people started buying televisions sets, and more broadcasters wanted to enter an area, the United States government stepped in and said that the diversity of systems wouldn’t fly - all television broadcasts and television sets had to be compatible.  To that end they created a new governing body, the National Television System Committee, or NTSC, which went on to define the first national television standard in 1941.

We’ve had to deal with the outcomes of standardization, good and bad, ever since.

The good, obviously, has been that we don’t have to buy a different television for every channel we want to watch, or every part of the country we want to live in (though transnationals are often still out of luck).  The bad is that every evolution of the standard since 1941 has required backwards compatibility: today’s digital broadcast standards, and computer display standards too, are still limited in part by what CRTs could do in the 1940s and 50s.

Don’t believe me?  Even ignoring the NTSC 1/1.001 frame rate modifier, there’s still a heavy influence: let’s look at the list:

  1. Color Space: The YIQ color space for NTSC and the YUV color space used in both PAL and SECAM are both based on the colors that can be produced by the short glow phosphors, which coat the inside of CRT screens and form the light and color producing element of the CRT.  In the transition to digital, YIQ and YUV formed the basis for Rec. 601 color space (SD Digital), which in turn is the basis for Rec. 709 (HD Digital) color space (Rec. 709 uses almost the same primaries as Rec. 601).

    And just in case your computer feels left out, the same color primaries are used in the sRGB display standard too, because all of these color spaces were display referenced, and they were all built on the same CRT technology.  Because up until the early 2000s, CRTs were THE way of displaying images electronically - LCDs were low contrast, plasma displays were expensive, and neither LEDs nor DLPs had come into their own.
     

  2. Transfer Function: The transfer function (also called the gamma curve) used in SD and HD is also based on the CRT’s natural light-to-electrical and electrical-to-light response.  The CRT camera captured images with a light-to-voltage response curve of approximately gamma 1/2.2, while the CRT display recreated images with a voltage-to-light response curve of approximately gamma 2.4.  Together, these values formed the standard approximate system gamma of 1.2, and form the basis for the current reference display gamma standard of 2.4, found in ITU-T Recommendation BT.1886.
     

  3. Brightness Limits: Lastly, and probably most frustratingly, color accurate CRT displays require limited brightness to maintain their color accuracy. Depending on the actual phosphors used for primaries, that max-brightness value typically lands in the 80-120 nits range.  And consumer CRT displays, while bigger, brighter, and less color accurate, still only land in the 200 nit max brightness levels.  For comparison, the brightness levels found on different outdoor surfaces during a sunny day land in the 5000-14,000 range (or more!).

    This large brightness disparity between reference and consumer display levels has been accentuated in recent years with the replacement of CRTs with LCD, Plasma and OLED displays, which can easily push 300-500 nits peak brightness.  Those brightness levels skew the overall look of images graded at reference, while being very intolerant of changes in ambient light conditions.  In short this means that with the current standards, consumers rarely have the opportunity to see content in their homes as filmmakers intended.

So, because of the legacy cathode ray tube, (a dead technology), we’re stuck with a set of legacy standards that limit how we can deliver images to consumers.  But because CRTs are a dead technology, we now have an opportunity where we can choose to either be shackled by the 1950s for the rest of time, or, to say “enough is enough,” and use something better.  Something forward thinking.  Something our current technology can’t even match 100% yet.  Something like, HDR video.


The HDR Way

At the moment, there two different categories and multiple standards covering HDR video, including CTA’s HDR 10 Media Profile, Dolby’s Dolby Vision, and the BBC’s Hybrid Log Gamma.  And naturally, they all do things just a little differently.  I’ll cover their differences in depth in Part 3: HDR Video Terms Explained, but for now I’m going to lump them all together and just focus on the common aspects of all HDR video, and what makes it different than video from the past.

There are four main things that are required to call something HDR video: ITU-T Recommendation BT.2020 or DCI-P3 color space, a high dynamic range transfer function, 10 bits per channel transmission and display values, and transmitted metadata.

BT.709, DCI-P3, and BT.2020 on CIE XYZ 1931

1. Color Space: For the most part, HDR video is seen by many as an extension of the existing BT.2020 UHD/FUHD and DCI specifications, and as such uses either the wider BT.2020 color gamut (BT.2020 is the 4K/8K replacement for BT.709/Rec.709 HD broadcast standards), or the more limited, but still wide, DCI-P3 gamut.

BT.2020 uses pure wavelength primaries, instead primary values based on the light emissions of CRT phosphors or any material.  The catch is, of course, we can’t fully show these in a desktop display (yet), and only the most recent laser projectors can cover the whole color range. But ultimately, the breadth of the color space covers as many of the visible colors as is possible with three real primaries*, and includes all color values already available in Rec.709/sRGB and DCI-P3, as well as 100% of Adobe RGB and most printer spaces available with today’s pigments and dyes.

2. Transfer Function: Where HDR video diverges from standard BT.2020 and DCI specs is in its light-level-to-digital-value and digital-value-to-light-level relationship, called the OETF and EOTF respectively.  I’m going to go into more depth on OETFs and EOTFs at another time, but for now what we need to know is that the current relationship between light levels and digital values is a legacy of the cathode ray tube days, and approximates gamma 2.4.  Under this system, full white digital value of 235 translates to a light output of between 80-120nits.

Extending this same curve into a higher dynamic range output proves problematic because of the non-linear response of the human eye: it would either cause severe stepping in the darks and lights, or it would require 14-16 bits per channel while wasting digital values in increments that can’t actually be seen.  And it still wouldn’t be backwards compatible, in which case, what’s the point?

So instead, HDR video uses one of two new transfer curves: the BBC’s Hybrid Log Gamma (HLG), standardized in ARIB STD-B67, which allows for output brightness levels from 0.01 nit up to around 5000 nits, and Dolby’s Perceptual Quantization (PQ) curve, standardized in SMPTE ST.2084, which allows for output brightness levels from 0.0001 nit up to 10,000 nits.

PQ is the result of direct research done by Dolby to measure the response of the human eye, and to create a curve where no value is wasted with no visible stepping between values.  The advantage of PQ is pretty clear, in terms of maximizing future output brightness (the best experimental single displays currently max out at 4000 nits; Dolby’s test apparatus ranged from 0.004 to 20,000 nits) and increasing the amount of detail captured in the darks.

HLG, on the other hand, provides a degree of backwards compatibility, matching the output levels of gamma 2.4 for the first 50% of the curve, and reserving the top 50% of the values to the higher light level output.  Generally, HLG content with a system gamma of 1.2 looks pretty close to standard dynamic range content, though it’s whites sometimes end up compressed and greyer than content mastered in SDR to begin with.

Footage graded in Rec. 709 and the same graded in HLG.

(Side note: I prefer grading in SMPTE ST.2084 because of the extended dynamic range through the blacks, and smoother roll-into the whites.)
 

3. Bit Depth: The new transfer curves accentuate a problem that’s been with video since the switch from analog to digital values: stepping.  As displays have gotten brighter, the difference between two code values (say, digital value of 25 and 26) is sometimes enough that we can see a clear distinguishing line between the two greys.  This is especially true when using a display whose maximum brightness is greater than reference standard, and is more common in the blacks than in the whites.

Both the BT.2020 and DCI standards already have requirements to decrease stepping by switching signal encoding and transmission from 8 bits per channel to 10 bits minimum (12 bits for DCI), allowing for at least a 4 times smoother gradient.  However, BT.2020 still permits 8 bit rendering at the display, which is what you’ll find on the vast majority of televisions and reference displays on the market today.

On the other hand, HDR video goes one step further and requires 10 bit rendering at the display panel itself; that is, each color sub pixel must be capable of between 876 and 1024 distinguishable light levels, in all operational brightness and contrast modes.

The reason that HDR requires a 10 bit panel while BT.2020 doesn’t, is that our eyes are more susceptible to stepping in the value of a color or gradient than to stepping in its hue or saturation: the eye can easily make up for lower color fidelity (8 bits per channel in BT.2020 space) by filling in the gaps, but with an HDR curve the jump in light levels between two codes in 8 bits per channel is big enough that it’s clearly noticeable.

Comparison between gradients step sizes at 8 bit, 10 bit, and 12 bit precisions (contrast emphasized)

4. Metadata: The last thing that HDR video requires that standard BT.2020 doesn’t, is metadata.  All forms of HDR video should include information about both the content and the mastering environment.  This includes which EOTF was used in the grade, the maximum and frame average brightnesses of the content and display, and which RGB primaries were used.  Dolby Vision even includes metadata to define, shot by shot, how to translate the HDR values into the SDR range!

Consumer display manufacturers use this information to adapt content for their screens in real time, knowing when to clip or compress the highlights and darks (based on the capability of the screen it’s being shown on), and for the automatic selection of operational mode (switching from Rec. 709 to BT.2020, and in and out of HDR mode, without the end user ever having to change a setting).

 

So, in summary, what does HDR video do differently?  Wider color gamuts, new transfer function curves to allow for a much larger range of brightnesses, 10 bits per channel minimum requirement at the display to minimize stepping, and the transmission of metadata to communicate information about the content and its mastering environment to the end user.

All of which are essential, none of which are completely backwards compatible.


Yes, but what does it look like?

Unfortunately, the only way to really show you what HDR looks like is to tell you to go to a trade show or post house with footage to show, or buy a TV with HDR capabilities and stream some actual HDR content.  Because when you show HDR content on a normal display, it does not look right:

Images in SMPTE ST.2084 HDR Video formats do not appear normal when directly brought into Rec. 709 or sRGB Gamma 2.4 systems

You can get a little bit of a feel for it if I cut the brightness levels of a standard dynamic range image by half, and put it side-by-side with one that more closely follows the HDR range of brightnesses:

Normalized & Scaled SMPTE ST.2084 HDR Video vs Rec. 709 with Brightness Scaled

But that doesn’t capture what HDR video actually does.  I don’t quite know how to describe it - it’s powerful, beautiful, clear, real, present and multidimensional.  There’s an actual physiological and psychological response to the image that you don’t get with standard dynamic range footage - not simply an emotional response to the quality of the image, but the higher brightness levels actually trigger things in your eyes and brain that let you literally see it differently than anything you’ve seen before.

And once you start using it on a regular basis, nothing else seems quite as satisfactory, no other image quite as beautiful.  You end up with a feeling that everything else is just a little bit inadequate.  That’s why HDR will very rapidly become the new normal of future video.


So that's it for Part 1: What is HDR Video?  In Part 2 of our series on HDR video, we’re going to cover what you need to grade in HDR, and how can you cheat a bit to get a feel for the format by emulating its response curve on your existing reference hardware.

Written by Samuel Bilodeau, Head of Technology and Post Production


Endnotes:

* While ACES does cover the entire visible color spectrum, it’s primary RGB values are imaginary, which means that while it can code for all possible colors, there’s no way of building a piece of technology that actually uses the ACES RGB values as its primary display colors.  Or in other words, if you were to try and display ACES full value RED, you couldn’t, because that color doesn’t exist.