Add a Color Format core option: 24-bit and 30-bit truecolor rendering

The software renderer wrote RGB565 unconditionally.  That is fine for the
palette's own colours but it quantises everything derived from them: the
light ramp in V_Palette16 carries 64 weights per colour and the Smooth
shading ramp 256, yet a 5-bit channel can only express about 17 distinct
levels of either, so distance shading and smooth gradients band.  Native
24/32-bit source art (PNG title cards, the help screen, the intermission
backdrop) was narrowed on the way to the surface for the same reason.

Add a "Color Format" core option selecting the output pixel format:

  16bits             RGB565        (default, the historical renderer)
  24bits (truecolor) XRGB8888
  30bits (HDR)       XRGB2101010

Both truecolor formats are native pipelines rather than a conversion
stage.  The palette (V_PaletteTC), the composed colour tables and the
Smooth ramp are all built at the output's channel width from the same
8-bit gamma-corrected palette source, so no value is ever produced in
565 and re-expanded, and no value passes through 8-bit on its way to
10-bit.  Measured on the shading ramp, a 256-step gradient resolves to
roughly 17 levels per channel at 16 bits, about 135 at 24 and close to
the full 256 at 30 -- the gap widening in dark colours, which is where
the banding is most visible.

The renderer twin lives in r_drawtc.c.  The column/span/wall-run drawers,
composed-LUT machinery and dispatch tables mirror r_draw.c exactly with a
16->32-bit surface retype; the two 32-bit formats share that single
instantiation, because an opaque drawer writes lut[texel] and is correct
for any format given a format-correct table.  Only the read-modify-write
blend kernels carry per-format channel constants, so only those are
compiled twice, from r_drawtcfmt.inl (SSE2 and NEON, with the scalar
forms as the bit-exact references, matching the existing 565 kernels).

Everything that wrote the surface directly is routed the same way: the
direct sprite column, the voxel rasteriser, the skybox scratch, the
plane and wall dynamic-light tints, the underwater tint, the wipe melt
and the ACS overlay compositors.  Several of those become simpler rather
than more complex -- the ACS alpha compositors previously expanded the
565 destination to 8 bits, blended, and re-narrowed on store; in
truecolor the blend just runs at the destination's own width.

SURFACE_PIXEL_DEPTH becomes a runtime value (vid_pixelbytes).  It was
never used in preprocessor arithmetic, so this is a straight
substitution; the cache and scratch buffers that size themselves from it
needed no other change.  The skybox scratch moves from a fixed 8 MB
16-bit static to a lazily sized allocation, which is smaller than before
in the 16-bit build.

Format selection happens once in retro_load_game, before any surface,
palette or table is built.  30-bit is gated on the new
RETRO_ENVIRONMENT_GET_SCREEN_10BPC_CAPABLE query rather than on
SET_PIXEL_FORMAT alone: SET_PIXEL_FORMAT accepts XRGB2101010
unconditionally and the frontend silently narrows the frame when the
driver cannot present 10 bits, and that narrowing truncates where our own
XRGB8888 path rounds.  So on a non-capable driver, emitting 10-bit is not
merely wasted work but measurably worse output than rendering 24-bit;
the core falls back to 24-bit and logs why.  An older frontend that does
not recognise the call returns false, which the contract defines as "no
guarantee of native 10-bit" -- the same branch.  A SET_PIXEL_FORMAT
rejection still degrades 30 -> 24 -> 16 underneath.

Note that XRGB2101010 here is deeper-precision SDR; the core emits no HDR
transfer function or metadata.

Also sync libretro-common with RetroArch f3fea37, which is where that
environment call comes from.  Only include/libretro.h differs from the
vendored copy; the other commits in that range touch image/mp4/webm
sources this core does not vendor.

The 16-bit path is untouched and stays the reference: frame-hash output
over Doom, Doom II, Heretic and zdcmp2.pk3 is byte-identical to the
parent commit.  24-bit and 30-bit were checked over the same content
through the title screen, the 3D scene, sprites, the status bar, the
intermission and the wipes.

Author: LibretroAdmin

Makefile.common

@@ -114,6 +114,8 @@ SOURCES_C += $(CORE_DIR)/am_map.c \
 				 $(CORE_DIR)/r_bsp.c \
 				 $(CORE_DIR)/r_data.c \
 				 $(CORE_DIR)/r_draw.c \
+				 $(CORE_DIR)/r_drawtc.c \
+				 $(CORE_DIR)/vid_mode.c \
 				 $(CORE_DIR)/r_main.c \
 				 $(CORE_DIR)/r_plane.c \
 				 $(CORE_DIR)/r_drawcmd.c \

README.md

@@ -186,12 +186,34 @@ the wad.
   them, with colours taken from the definition or derived from the texture
   itself.
 
+### Colour depth
+
+The **Color Format** core option selects the output pixel format: `16bits`
+(RGB565, the default and the historical renderer), `24bits (truecolor)`
+(XRGB8888) or `30bits (HDR)` (XRGB2101010). The truecolor formats are native
+pipelines, not a conversion stage — the palette, light ramps and composed
+colour tables are built at the output's channel width, so nothing is
+quantised to 565 on the way. This mainly removes the banding the 16-bit
+light ramp introduces in distance shading and in smooth gradients: a
+256-step shading ramp resolves to roughly 17 levels per channel at 16 bits,
+about 135 at 24, and near the full 256 at 30 — the gap being widest in dark
+colours, where the banding is most visible. Full-colour art (PNG title
+cards, the help screen, the intermission backdrop) is also blitted without
+losing precision.
+
+`30bits` is used only when the frontend reports that it can present a
+10-bit surface natively; otherwise the core falls back to `24bits`, which
+rounds rather than truncates and so looks better than a narrowed 10-bit
+frame. Note that XRGB2101010 is deeper-precision SDR — the core emits no
+HDR transfer function or metadata.
+
+Changing the option requires a restart.
+
 ### Not supported
 
 - **ZScript** — no support. Mods whose gameplay lives in ZScript won't run it.
 - **Line / sector portals** — inert.
 - **3D models (MODELDEF)** — out of scope for the 8-bit software renderer.
-- **Truecolor rendering.**
 
 The practical result: map-and-resource-driven ZDoom wads — new levels, sprite
 and texture replacements, ACS-scripted set pieces, reskinned monsters, Hexen-

libretro/libretro-common/include/libretro.h

@@ -2715,6 +2715,34 @@ enum retro_mod
  */
 #define RETRO_ENVIRONMENT_SET_SERIALIZATION_QUIRKS 87
 
+/**
+ * Queries whether the active video driver can present a 10-bit-per-channel
+ * (30-bit) source surface end to end, i.e. whether a frame submitted as
+ * #RETRO_PIXEL_FORMAT_XRGB2101010 reaches the display without the frontend
+ * narrowing it to 8 bits per channel.
+ *
+ * Unlike SET_PIXEL_FORMAT, which accepts XRGB2101010 unconditionally and
+ * transparently down-converts when the driver cannot present 10-bit, this
+ * lets a core discover the real capability so it can avoid pointless work:
+ * a core that has both a 10-bit and an 8-bit output path should prefer the
+ * 8-bit path when this returns \c false, since emitting 10-bit only to have
+ * the frontend narrow it wastes effort and, for content that starts at 8
+ * bits, is a no-op round trip; going straight to 8 bits also rounds rather
+ * than truncates.
+ *
+ * The result may change across a driver reinit (e.g. the user switches
+ * video driver or toggles HDR), so a core that cares should query it when
+ * (re)choosing its pixel format rather than caching it indefinitely.
+ *
+ * @param[out] data <tt>bool *</tt>.
+ * Set to \c true if a 10-bit source surface is presented natively, \c false
+ * if XRGB2101010 frames are down-converted to 8-bit.
+ * @return \c true if the environment call is recognised (the value at
+ * \c data is then valid), \c false if it is unsupported (an older frontend);
+ * a core must treat "unsupported" as "no guarantee of native 10-bit".
+ */
+#define RETRO_ENVIRONMENT_GET_SCREEN_10BPC_CAPABLE (88 | RETRO_ENVIRONMENT_EXPERIMENTAL)
+
 /**
  * Result of \c RETRO_ENVIRONMENT_GET_MEMORY_STATUS.
  *

libretro/libretro.c

@@ -31,6 +31,7 @@
 #endif
 
 #include "libretro_core_options.h"
+#include "../src/vid_mode.h"
 
 /* prboom includes */
 
@@ -936,32 +937,10 @@ void retro_init(void)
          midi_iface_valid = false;
    }
 
-   /* Negotiate pixel format with the frontend.  The renderer is
-    * hardcoded to write 16-bit pixels via VID_PAL16 -- the
-    * surface depth is fixed at SURFACE_PIXEL_DEPTH=2 and every
-    * draw column / span helper writes uint16_t directly.  We
-    * cannot fall through to XRGB1555 silently if RGB565 is
-    * rejected; the output would be unspecified-format garbage
-    * (with the wrong red/blue channel widths and an alpha bit
-    * smeared across the green LSB).  Log a hard error so the
-    * user sees a visible diagnostic instead of a corrupted
-    * screen. */
-   {
-      enum retro_pixel_format rgb565 = RETRO_PIXEL_FORMAT_RGB565;
-      if (!environ_cb(RETRO_ENVIRONMENT_SET_PIXEL_FORMAT, &rgb565))
-      {
-         if (log_cb)
-            log_cb(RETRO_LOG_ERROR,
-                   "Frontend rejected RGB565 pixel format -- this "
-                   "core requires it.  Update RetroArch or your "
-                   "frontend to a build that supports RGB565.\n");
-      }
-      else if (log_cb)
-      {
-         log_cb(RETRO_LOG_DEBUG,
-                "Frontend accepted RGB565 pixel format.\n");
-      }
-   }
+   /* Pixel format is negotiated in retro_load_game, once the "Color
+    * Format" core option has been read -- it must be settled before any
+    * surface, palette or lookup table is built, and retro_init runs before
+    * the frontend can answer GET_VARIABLE. */
 
    if (environ_cb(RETRO_ENVIRONMENT_GET_INPUT_BITMASKS, NULL))
       libretro_supports_bitmasks = true;
@@ -1314,6 +1293,112 @@ static void update_audio_samplerate(void)
    I_SetSoundRate(chosen);
 }
 
+/*
+ * I_NegotiatePixelFormat
+ *
+ * Settle the output pixel format for the whole session, from the "Color
+ * Format" core option.  Must run before update_variables() computes
+ * SCREENPITCH and before any surface, palette or lookup table is built --
+ * SURFACE_PIXEL_DEPTH is derived from the result.
+ *
+ * The 30-bit path is gated on RETRO_ENVIRONMENT_GET_SCREEN_10BPC_CAPABLE
+ * rather than on SET_PIXEL_FORMAT alone.  SET_PIXEL_FORMAT accepts
+ * XRGB2101010 unconditionally and the frontend silently narrows the frame
+ * to 8 bits when the active video driver cannot present a 10-bit surface,
+ * so acceptance says nothing about what reaches the display.  Worse, that
+ * narrowing truncates, while our own XRGB8888 path rounds -- so on a
+ * non-capable driver the 10-bit renderer is not merely wasted work, it is
+ * measurably worse output than just rendering 24-bit.  Query the real
+ * capability and drop to 24-bit when it is not there.
+ *
+ * An older frontend that does not recognise the call returns false, which
+ * the libretro contract defines as "no guarantee of native 10-bit" -- the
+ * safe default, and the same branch we take for a non-capable driver.
+ */
+static void I_NegotiatePixelFormat(void)
+{
+   struct retro_variable var;
+   enum retro_pixel_format fmt;
+   int want = VID_MODE565;
+
+   var.key   = "prboom-color_format";
+   var.value = NULL;
+   if (environ_cb(RETRO_ENVIRONMENT_GET_VARIABLE, &var) && var.value)
+   {
+      if (!strcmp(var.value, "24bits (truecolor)"))
+         want = VID_MODE8888;
+      else if (!strcmp(var.value, "30bits (HDR)"))
+         want = VID_MODE2101010;
+   }
+
+   if (want == VID_MODE2101010)
+   {
+      bool tenbit = false;
+      if (!environ_cb(RETRO_ENVIRONMENT_GET_SCREEN_10BPC_CAPABLE, &tenbit)
+            || !tenbit)
+      {
+         want = VID_MODE8888;
+         if (log_cb)
+            log_cb(RETRO_LOG_INFO,
+                   "Color Format: 30-bit requested, but the frontend does "
+                   "not present a 10-bit source natively -- using 24-bit "
+                   "truecolor instead (rounds rather than truncates).\n");
+      }
+   }
+
+   /* Ask for the chosen format, degrading if the frontend refuses it. */
+   for (;;)
+   {
+      if (want == VID_MODE2101010)
+         fmt = RETRO_PIXEL_FORMAT_XRGB2101010;
+      else if (want == VID_MODE8888)
+         fmt = RETRO_PIXEL_FORMAT_XRGB8888;
+      else
+         fmt = RETRO_PIXEL_FORMAT_RGB565;
+
+      if (environ_cb(RETRO_ENVIRONMENT_SET_PIXEL_FORMAT, &fmt))
+         break;
+
+      if (want == VID_MODE2101010)
+      {
+         want = VID_MODE8888;
+         if (log_cb)
+            log_cb(RETRO_LOG_WARN,
+                   "Frontend rejected XRGB2101010; falling back to "
+                   "XRGB8888.\n");
+      }
+      else if (want == VID_MODE8888)
+      {
+         want = VID_MODE565;
+         if (log_cb)
+            log_cb(RETRO_LOG_WARN,
+                   "Frontend rejected XRGB8888; falling back to RGB565.\n");
+      }
+      else
+      {
+         /* RGB565 refused: the renderer cannot produce anything else that
+          * is guaranteed, so log loudly rather than emitting pixels in an
+          * unspecified format (wrong channel widths, smeared alpha). */
+         if (log_cb)
+            log_cb(RETRO_LOG_ERROR,
+                   "Frontend rejected RGB565 pixel format -- this core "
+                   "requires it.  Update RetroArch or your frontend to a "
+                   "build that supports RGB565.\n");
+         break;
+      }
+   }
+
+   vid_mode       = want;
+   vid_pixelbytes = (want == VID_MODE565) ? 2 : 4;
+
+   if (log_cb)
+      log_cb(RETRO_LOG_INFO, "Color Format: %s (%d bytes/pixel).\n",
+             (want == VID_MODE2101010) ? "30-bit XRGB2101010"
+             : (want == VID_MODE8888)  ? "24-bit XRGB8888"
+                                       : "16-bit RGB565",
+             vid_pixelbytes);
+}
+
 static void update_variables(bool startup)
 {
    struct retro_variable var;
@@ -2011,6 +2096,11 @@ bool retro_load_game(const struct retro_game_info *info)
    else if (log_cb)
       log_cb(RETRO_LOG_INFO, "Rumble environment not supported.\n");
 
+   /* Settle the pixel format first: update_variables() derives SCREENPITCH
+    * from SURFACE_PIXEL_DEPTH, and everything allocated below is sized by
+    * it. */
+   I_NegotiatePixelFormat();
+
    update_variables(true);
 
    argv[argc++] = strdup("prboom");
@@ -3587,14 +3677,20 @@ dbool   I_StartDisplay(void)
       fb.width        = SCREENWIDTH;
       fb.height       = SCREENHEIGHT;
       fb.pitch        = 0;
-      fb.format       = RETRO_PIXEL_FORMAT_RGB565;
+      fb.format       = (vid_mode == VID_MODE2101010)
+                        ? RETRO_PIXEL_FORMAT_XRGB2101010
+                        : (vid_mode == VID_MODE8888)
+                          ? RETRO_PIXEL_FORMAT_XRGB8888
+                          : RETRO_PIXEL_FORMAT_RGB565;
       fb.access_flags = RETRO_MEMORY_ACCESS_WRITE;
       fb.memory_flags = 0;
 
+      {
+      const enum retro_pixel_format want_fmt = fb.format;
       if (environ_cb(RETRO_ENVIRONMENT_GET_CURRENT_SOFTWARE_FRAMEBUFFER,
                      &fb)
             && fb.data
-            && fb.format == RETRO_PIXEL_FORMAT_RGB565
+            && fb.format == want_fmt
             && fb.pitch  == (size_t)SCREENPITCH)
       {
          /* Repoint screens[0] and the renderer's cached top-left
@@ -3613,11 +3709,11 @@ dbool   I_StartDisplay(void)
           * to copy into the differently-strided destination
           * itself.
           *
-          * Format must be RGB565 -- frontends are allowed to
-          * return a different format than SET_PIXEL_FORMAT
-          * negotiated, e.g. when running with a HW backend that
-          * needs an internal conversion stage.  We can only
-          * direct-render when the formats agree. */
+          * Format must match the one we negotiated -- frontends
+          * are allowed to return a different format than
+          * SET_PIXEL_FORMAT settled on, e.g. when running with a
+          * HW backend that needs an internal conversion stage.
+          * We can only direct-render when the formats agree. */
          direct_fb_data         = (unsigned char *)fb.data;
          direct_fb_pitch        = (unsigned int)fb.pitch;
          screens[0].data        = direct_fb_data;
@@ -3634,6 +3730,7 @@ dbool   I_StartDisplay(void)
             have_sw_fb    = true;
          }
       }
+      }
    }
 
    return true;

libretro/libretro_core_options.h

@@ -61,6 +61,21 @@ struct retro_core_option_v2_definition option_defs_us[] = {
       },
       "320x200"
    },
+   {
+      "prboom-color_format",
+      "Color Format (Restart Required)",
+      NULL,
+      "Output colour depth. '16bits' is the classic RGB565 renderer. '24bits' renders in full 8-bit-per-channel truecolor, which removes the banding the 16-bit light ramp introduces in distance shading and smooth gradients. '30bits' renders at 10 bits per channel for finer gradients still; it is used only when the frontend can actually present a 10-bit surface, otherwise the core falls back to 24bits automatically.",
+      NULL,
+      NULL,
+      {
+         { "16bits",             NULL },
+         { "24bits (truecolor)", NULL },
+         { "30bits (HDR)",       NULL },
+         { NULL, NULL },
+      },
+      "16bits"
+   },
    {
       "prboom-mouse_on",
       "Mouse Active When Using Gamepad",

src/config.h

@@ -10,7 +10,14 @@
 #include <compat/msvc.h>
 #endif
 
-#define SURFACE_PIXEL_DEPTH 2
+/* Bytes per surface pixel.  Runtime, not a constant: the "Color Format"
+ * core option selects RGB565 (2) or a 32-bit truecolor format (4) once at
+ * load, before any surface is allocated.  vid_pixelbytes lives in
+ * vid_mode.c and stays 2 until the libretro layer negotiates otherwise, so
+ * every consumer that predates the option behaves exactly as before.
+ * Only used in ordinary expressions -- never in preprocessor arithmetic. */
+extern int vid_pixelbytes;
+#define SURFACE_PIXEL_DEPTH vid_pixelbytes
 extern int SCREENWIDTH;
 extern int SCREENHEIGHT;
 extern int SCREENPITCH;

src/f_wipe.c

@@ -135,10 +135,14 @@ static int wipe_doMelt(int ticks)
        * screen rather than relying on previous frames' writes. */
       for (y = 0; y < SCREENHEIGHT; y++)
       {
-         uint16_t *drow =
-            (uint16_t *)(wipe_scr.data + (size_t)y * SURFACE_BYTE_PITCH);
-         const uint16_t *erow =
-            (const uint16_t *)(wipe_scr_end.data + (size_t)y * SURFACE_BYTE_PITCH);
+         /* Byte-addressed: the melt is a pure copy with no colour maths,
+          * so the same loop serves every surface width -- pixel offsets
+          * just scale by SURFACE_PIXEL_DEPTH.  Nothing here converts
+          * between formats. */
+         uint8_t *drow =
+            wipe_scr.data + (size_t)y * SURFACE_BYTE_PITCH;
+         const uint8_t *erow =
+            wipe_scr_end.data + (size_t)y * SURFACE_BYTE_PITCH;
 
          i = 0;
          while (i < SCREENWIDTH)
@@ -157,18 +161,21 @@ static int wipe_doMelt(int ticks)
                      break;
                   run++;
                }
-               memcpy(&drow[i], &erow[i],
+               memcpy(drow + (size_t)i * SURFACE_PIXEL_DEPTH,
+                      erow + (size_t)i * SURFACE_PIXEL_DEPTH,
                       (size_t)(run - i) * SURFACE_PIXEL_DEPTH);
                i = run;
             }
             else
             {
                /* Start-screen pixel, scrolled down by the boundary:
                 * dest row y reads source row y - b. */
-               const uint16_t *srow =
-                  (const uint16_t *)(wipe_scr_start.data
-                     + (size_t)(y - b) * SURFACE_BYTE_PITCH);
-               drow[i] = srow[i];
+               const uint8_t *srow =
+                  wipe_scr_start.data
+                     + (size_t)(y - b) * SURFACE_BYTE_PITCH;
+               memcpy(drow + (size_t)i * SURFACE_PIXEL_DEPTH,
+                      srow + (size_t)i * SURFACE_PIXEL_DEPTH,
+                      SURFACE_PIXEL_DEPTH);
                i++;
             }
          }