bug: ffmpeg decoding of dnx36,prores, etc. not correct

[groucho86]

Extracting an image from DNxHD or ProRes and saving it as a TIFF, PNG, etc. is darker than expected. This matches ffmpeg's default behavior when a -vf is not explicitly set.

% oiiotool --buildinfo OIIO 3.1.8.0 | MacOS/ARM Build compiler: Apple clang 17.0 | C++17/201703 HW features enabled at build: neon No CUDA support (disabled / unavailable at build time) Dependencies: BZip2 1.0.8, FFmpeg 8.0, fmt 12.1.0, Freetype 2.14.1, GIF 5.2.2, Imath 3.2.2, JXL 0.11.1, Libheif 1.20.2, libjpeg-turbo 3.1.2, LibRaw 0.21.4, libuhdr NONE, OpenColorIO 2.5.0, OpenEXR 3.4.4, openjph 0.25.3, OpenVDB NONE, PNG 1.6.51, pugixml 1.15, pybind11 3.0.1, Python3 3.14.0, Robinmap 1.4.0, TBB 2022.3.0, TIFF 4.7.1, WebP 1.6.0, ZLIB 1.3.1

To Reproduce

WRONG OUTPUT:
oiiotool --debug -v --info -i /tmp/imagery_dnxhd36.mov --subimage 1 -o /tmp/oiio_png_from_imagery_dnxhd36.png

Which matches

ffmpeg -v debug -i /tmp/imagery_dnxhd36.mov -vframes 1 -pix_fmt rgb24 -y /tmp/ffmpeg_implicit_png_from_imagery_dnxhd36.png

CORRECT OUTPUT (with explicit color scaling operations):
ffmpeg -v debug -i /tmp/imagery_dnxhd36.mov -vf "colorspace=all=bt709:range=pc:iall=bt709:irange=tv" -vframes 1 -pix_fmt rgb24 -y /tmp/ffmpeg_explicit_png_from_imagery_dnxhd36.png

Attaching sample legal range DNxHD 36 (exported out of Resolve)
https://github.com/user-attachments/assets/81027a1a-6bcc-454b-b66a-6ff6ae0f26a2

[zachlewis]

Ah ha... so, the problem is, OpenImageIO isn't scaling narrow range video levels to full range as expected.

I think it's reasonable to expect OIIO readers that support CICP to automatically convert to full range when CICPRange == 0 (when the video full range flag is off); and maybe a global attribute for disabling that behavior? I believe Nuke Read nodes have a similar "Convert Video Range to Full Range" toggle that's enabled by default.

[lgritz]

Yes, OIIO convention is that on the app-side of the API, only 8-, 16- and 32-bit data types are supported, and for unsigned integer types, always assuming the full range represents the 0-1 scale. Any "in between" bit depths read from a file are rounded up to the next bigger type and scaled to take up the full range. For example, a TIFF file with 12 bits per sample data (0-4095) looks like a 16 bit TIFF file (0-65535), but an extra metadata hint "oiio:BitsPerSample" is set to 12 as a hint that this is how it was stored in the file. That same hint, upon output, is taken as a request to please try to write with that bit depth (with automatic range rescaling) if the output format supports such a thing.

[groucho86]

To be clear, this is not a "ffmpeg is misinterpreting the source as full range". The mismatch isn't that drastic.

According to claude.ai (apologies...), the issue is swscale is combining the scaling+matrix operation "in one integer math pass", compounding the errors. By separating the two operations, it reduces the accumulated errors.

Claude.ai offered a minimal patch for ffmpeginput.cpp but I don't know C++ and don't want to pollute this great library with potential AI-slop. Happy to copy-paste the suggestion here though.

[zachlewis]

To be clear, this is not a "ffmpeg is misinterpreting the source as full range". The mismatch isn't that drastic.

Ah...! I see exactly what you're talking about. Great catch! And thank you for providing such explicit reproduction instructions and a reference video, that's really helpful.

the issue is swscale is combining the scaling+matrix operation "in one integer math pass", compounding the errors. By separating the two operations, it reduces the accumulated errors.

That makes sense. Maybe instead of separating the two operations, we can get away with coercing higher-precision math...

Can you confirm that the following ffmpeg command works at least as well as the explicit range scaling command?

ffmpeg -v debug -i /tmp/imagery_dnxhd36.mov  -sws_flags accurate_rnd+full_chroma_int+full_chroma_inp   -vframes 1 -pix_fmt rgb24 -y /tmp/sws_flags.png

If so, I think we should do likewise in our ffmpeg reader.

src/ffmpeg.imageio/ffmpeginput.cpp, ~line 498:

 m_sws_rgb_context
        = sws_getContext(m_codec_context->width, m_codec_context->height,
                         src_pix_format, m_codec_context->width,
                         m_codec_context->height, m_dst_pix_format,
                         SWS_ACCURATE_RND | SWS_FULL_CHR_H_INT | SWS_FULL_CHR_H_INP,
                         NULL, NULL, NULL);

The SWS flags we're setting here are:

• accurate_rnd: higher-precision operations
• full_chroma_inp: chroma interpretation: treat inputs chroma samples as full resolution horizontally during conversion (e.g., for YUV422, where chroma is sampled at half horizontal resolution).
• full_chroma_int: chroma interpolation: better edges & saturation with YUV420 and YUV422

(note, I'm getting rid of our current use of the SWS_AREA flag here -- "area" refers to swscale's resampling kernel, and our reader isn't doing any resizing)

Optionally, there's a bitexact flag we could set -- it's a little slower, because it disables platform-specific optimizations, but it produces deterministic outputs across platforms. I don't think it's something we should enable by default, but it's worth mentioning, in case this causes CI trouble.

Claude.ai offered a minimal patch for ffmpeginput.cpp [...] Happy to copy-paste the suggestion here

Sure, I'd be curious to know what Claude suggests.

[groucho86]

Can you confirm that the following ffmpeg command works at least as well as the explicit range scaling command?

I can confirm it works at least as well. Neither 2 methods match as accurately as what Resolve does, but that's probably a topic for another day/repo :)

Sure, I'd be curious to know what Claude suggests.

This is what claude suggested:

+++ ffmpeginput.cpp	2025-12-23 00:00:00.000000000 +0000
@@ -496,6 +496,45 @@
 
     m_sws_rgb_context
         = sws_getContext(m_codec_context->width, m_codec_context->height,
                          src_pix_format, m_codec_context->width,
                          m_codec_context->height, m_dst_pix_format, SWS_AREA,
                          NULL, NULL, NULL);
+
+    // Configure proper color space conversion for legal range video
+    // Get the source color properties from the codec context
+    int src_range = (m_codec_context->color_range == AVCOL_RANGE_JPEG) ? 1 : 0;  // 0=limited/TV, 1=full/PC
+    int dst_range = 1;  // Always output full range RGB
+    
+    // Map FFmpeg colorspace to swscale coefficients
+    // FFmpeg AVCOL_SPC values match ITU-T H.273 / CICP
+    int src_colorspace;
+    switch (m_codec_context->colorspace) {
+        case AVCOL_SPC_BT709:
+            src_colorspace = SWS_CS_ITU709;
+            break;
+        case AVCOL_SPC_BT470BG:
+        case AVCOL_SPC_SMPTE170M:
+            src_colorspace = SWS_CS_ITU601;
+            break;
+        case AVCOL_SPC_SMPTE240M:
+            src_colorspace = SWS_CS_SMPTE240M;
+            break;
+        case AVCOL_SPC_BT2020_NCL:
+        case AVCOL_SPC_BT2020_CL:
+            src_colorspace = SWS_CS_BT2020;
+            break;
+        default:
+            // Default to BT.709 for HD content, BT.601 for SD
+            src_colorspace = (m_codec_context->height >= 720) ? SWS_CS_ITU709 : SWS_CS_ITU601;
+            break;
+    }
+    
+    // Set colorspace details: converts YUV (with proper matrix/range) to RGB full range
+    const int* inv_table = sws_getCoefficients(src_colorspace);
+    const int* table = sws_getCoefficients(SWS_CS_DEFAULT);  // RGB doesn't need a matrix
+    
+    sws_setColorspaceDetails(m_sws_rgb_context,
+                             inv_table, src_range,
+                             table, dst_range,
+                             0, 1 << 16, 1 << 16);
 
     AVDictionaryEntry* tag = NULL;
     while ((tag = av_dict_get(m_format_context->metadata, "", tag,

[zachlewis]

If you set the pix_fmt to rgb48le instead of rgb24, do you get results closer to what you're seeing in Resolve?

[groucho86]

rgb48le helps. But still, neither match as accurately as what Resolve (and Adobe Media Encoder) offer.

The following ffmpeg flags are "fine":

-vf "colorspace=all=bt709:range=pc:iall=bt709:irange=tv"
-sws_flags lanczos+accurate_rnd+full_chroma_int+full_chroma_inp+bitexact

The most accurate Y'CbCr to RGB ffmpeg conversion requires z.lib instead of swscale:
-vf "zscale=rin=limited:r=full:t=bt709:m=bt709:p=bt709:dither=error_diffusion,format=gbrp16le"

(format=gbrp16le is critical)

[lgritz]

@zachlewis We let this slip through the cracks for a while, but did you have something cooking that addresses this issue?

[zachlewis]

Yes indeed, and it also addresses an issue with the recent PR that Shane submitted the other month for P3 support in JXL.

The issue here is, converting from narrow YUV to full RGB expands values beyond the normalized domain of the container, so having format readers handle both at the same time in order to hand off full-range RGB values at that same bitdepth means clipping "sub-black" and "super-white" values afforded by the encoding scheme. Even though these values are clipped by displays anyway during broadcast, that highlight detail is recoverable if the conversion to RGB takes place at a higher bitdepth.

(This is strictly a video thing -- JPEG's YUV encodings are full-range to begin with, so there's nothing lost by converting to 8-bit full-range RGB data.)

Long story short, we need a better strategy for handling YUV encodings... some ideas:

• Immediately convert to a float buffer for the YUV-to-RGB conversion
• Defer the conversion to RGB to the IBAs / output writers who are expecting to work with full-range RGB data, using CICP metadata to track the state of YUV-encoded ImageBufs, in order convey to downstream IBAs and format writers what more needs to be done to convert to a full-range RGB encoding.

Either way, my plan is to include narrow-range RGB and YUV color spaces in a builtin OCIO config, so whenever a color space conversion does take place, it takes place within the OCIO graph at the output bitdepth...

More soon! I've got a whole other PR I'm trying desperately to get to you before I can implement the CICP-in-OCIO thing fully.