RFC: Deprecate and ultimately remove the HLSL front-end

[dnovillo]

Back in 2022, @dj2 floated the idea that the HLSL front-end is not worth maintaining anymore (#2971). At the time, the conclusion was that maintenance is not a huge burden and there was objections from users deploying HLSL compilers in environments such as iOS, Android and tvOS.

In the last 4 years, DXC has come a long way and the language has evolved way beyond what glslang supports today.

I want to bring up this discussion up again, with the following proposal:

1. Formally mark the HLSL front-end as deprecated in the README and release notes.
2. Direct users to DXC (https://github.com/microsoft/DirectXShaderCompiler) and Slang (https://github.com/shader-slang/slang) as alternatives.
3. Continue accepting bug fixes during the deprecation period.
4. Remove the front-end at the next major version, with at least 18 months of notice from the deprecation announcement.

What would break: Any caller using EShSourceHlsl or the HLSL message flags (EShMsgReadHlsl, etc.) would need to migrate. This is a breaking API change and is the reason a major version bump is required.

I would like to avoid having to deal with security issues in the HLSL front-end and any other bugs that may require paying attention to a language that is better served by the current alternatives (mostly DXC, but clang-dxc in the medium/long-term).

Does this sound reasonable? I'm happy to do the removal work if there is agreement on the plan. I would love to get rid of O(20,000) lines of code that are not adding a lot of value to this compiler.

@TheMostDiligent is DXC still not a valid replacement for your use-case? I saw the analysis you posted in #2971 (comment) and I'm also not sure whether the issues reported by AnKi in https://anki3d.org/workarounds-for-issues-with-mesh-shaders-vulkan-hlsl/ are still present.

Thanks. Diego.

NOTE: Tagging folks who participated in previous discussions I read while preparing this: @dj2 @TheMostDiligent @dneto0 @arcady-lunarg @dgkoch @godlikepanos

[TheMostDiligent]

For Diligent Engine, removing HLSL support from glslang would be a show-stopper.

At this point, I do not see a viable replacement for our use case other than maintaining a fork of glslang with HLSL support, which would be an undesirable outcome for both us and the ecosystem.

DXC may support more modern HLSL features, but in practice it is primarily an offline compiler. That does not solve the class of problems we need to solve. Runtime shader compilation remains a viable and, in many scenarios, indispensable capability. In real applications, shader permutations are often determined by runtime configuration, device capabilities, material systems, user content, or feature combinations that are impractical to enumerate and precompile ahead of time. In those cases, asynchronous runtime compilation combined with caching is not a niche workflow; it is the right engineering solution.

glslang is particularly important to us because it enables that workflow in a way the alternatives currently do not:

1. It is relatively small and easy to embed without a major increase in application size or dependency footprint.
2. It is portable and works across the platforms we care about.
3. It is suitable for environments where shipping a large compiler stack is not practical.

Platform coverage is especially important for us. Android, iOS, tvOS, and Web are not edge cases in this discussion. For Web in particular, deployment constraints are strict enough that using a heavier compiler toolchain is often impractical.

More broadly, deprecating glslang’s HLSL front-end would not simply “direct users to better alternatives.” It would remove an important portability layer and eliminate a lightweight embedded runtime compilation option that some engines still depend on.

Even if the front-end is incomplete relative to DXC, that alone is not sufficient reason to remove it. A smaller, less feature-complete implementation still provides substantial value when it is portable, embeddable, and good enough for production workloads that need cross-platform runtime compilation.

From our perspective, the question is not whether DXC is more advanced. It clearly is. The question is whether it is an adequate replacement for all existing glslang HLSL users. For us, the answer today is no.

[dneto0]

For context, I presented the tradeoffs between Glslang and DXC for HLSL compilation, at GDC 2018. (About 32 minutes into https://gdcvault.com/play/1024853/What-s-New-in-Vulkan )

Broadly, I am in favour of the plan.

I maintain Shaderc, which wraps around Glslang, and offers both GLSL and HLSL compilation paths. I don't know how important the HLSL path is to my users. (I don't collect metrics) I would follow a deprecation plan that mirrors Glslang.

However, I'm facing a backlog of dozens of crashes and memory safety issues in Glslang's HLSL front end. It is a liability to me.

The HLSL backend is essentially unmaintained. It is not evolving. Because it is not changing, then working from a fork would be viable.

[TheMostDiligent]

Is it an option to keep HLSL path in the codebase, but unmaintained (or community-maintained?)

[dnovillo]

Is it an option to keep HLSL path in the codebase, but unmaintained (or community-maintained?)

The maintenance issue we have accumulated is also compounded by runtime compilation because the input is typically less controlled than offline precompilation. PRs #4207 and #4208 are recent examples of latent crash bugs that had been present for years. I am not confident there are not more.

One of the main issues with this, in my mind, is the friction it creates for GLSL itself. The HLSL front-end is not cleanly isolated. In addition to the ~20k LoC in glslang/HLSL/, HLSL-specific logic is embedded throughout the shared infrastructure:

• 25 compile-time gates (#ifdef ENABLE_HLSL) in shared headers and implementation files.
• 49 runtime language checks (EShSourceHlsl) in shared files outside of glslang/HLSL/. These are run-time branches inside functions that also serve GLSL, meaning every caller of those functions implicitly depends on understanding both code paths. The 49 runtime checks are not uniform. They cover semantically distinct behaviors that GLSL and HLSL handle differently. Each one represents a case where a maintainer modifying the shared code must reason about two languages simultaneously.
• Short-circuit evaluation. In GLSL, both if/else and the ternary operator ?: short-circuit. In HLSL, ?: does not. The TIntermSelection AST node carries a noShortCircuit flag set by the HLSL path. Any change to how addSelection builds the AST or how GlslangToSpv.cpp lowers it must account for this flag.
• Type conversion rules. HLSL's conversion rules for scalars, vectors, and matrices differ fundamentally from GLSL. HLSL allows a scalar to replicate into any matrix, a vector to truncate into a smaller vector, and float4 to be reinterpreted as a 2x2 matrix. A change to how GLSL implicit conversions work requires the author to ensure it remains correct for both.
• Bool-to-int promotion. HLSL implicitly promotes bool to int for 14 operator types (+, -, *, /, %, <<, >>, &, |, ^, <, >, <=, >=). GLSL does not.
• Version bypass. In GLSL, certain conversions (int-to-uint) require version 400 or higher. These checks contain || getSource() == EShSourceHlsl to bypass the restriction for HLSL. Any future GLSL version gating added to the same function must not accidentally remove the HLSL bypass.
• Implicit this parameter. HLSL member functions pass this as an implicit first parameter. This is guarded with #ifdef ENABLE_HLSL in the SPIR-V generator's parameter-handling loop, which runs for every function call in every shader. The variable implicitThis is set to false unconditionally without HLSL, but the guard must be preserved on any refactor of the handling code.
• I/O mapping. HLSL uses register spaces for binding (register(b0, space1)) rather than GLSL's explicit bindings. The TDefaultHlslIoResolver class is compiled into iomapper.cpp under ENABLE_HLSL. Any work on the I/O mapper must account for this class.

For the moment, if we could formally deprecate HLSL front end with a long deprecation window (say 18 to 24 months), that would give the community time to either fork HLSL support or re-implement it in a way that satisfies the embeddability requirements for the compiler.

[dj2]

If the HLSL backend isn't really changing, could you just tag it and remove it from main now? Anyone that wants the deprecated HLSL can work off the tag as they migrate to something new?

[TheMostDiligent]

The HLSL front-end is not cleanly isolated.

Is it possible to improve the isolation to make GLSL/HLSL isolation cleaner? Or that would be too much effort?

[dj2]

From memory of trying to do this 4 years ago, it would be really difficult. There are HLSL hooks everywhere, from deep in the preprocessor all the way through the rest of the stack. Some is easy to tell with ifdefs some is tied to options and arguments. It's really hard to untangle it.

[dnovillo]

Thank you for the discussion. Here is my proposal for next steps:

1. Mark the HLSL front-end as deprecated (Deprecate the HLSL front-end. #4217)
2. Create an annotated tag (hlsl-last-supported) at that commit as a stable branch point for anyone who needs to fork.

What I'm not proposing to change yet

Removing the front-end is more work than marking it deprecated. The HLSL path is not cleanly isolated: there are several HLSL bits in shared code paths with GLSL. @dj2 noted from prior experience that improving this isolation would be very difficult. I'm not planning to address that here.

My intention is to remove the front-end at the next major version, with at least 18 months from this deprecation announcement.

Proposed policy for the deprecation period

• Incoming bug reports against the HLSL front-end would be closed with a pointer to this issue.
• Security reports would be assessed individually, but I would not plan to fix complex ones.
• Low-risk PRs fixing HLSL bugs without touching shared infrastructure could still be accepted if they include tests and a contributor willing to review them.
• No new HLSL features.
• GLSL improvements to shared code take precedence over HLSL compatibility.

@TheMostDiligent, I understand your use case and agree it is not served by DXC or Slang today. I don't think we can maintain that in the upstream project, but a fork at the hlsl-last-supported tag is an option.

The maintenance cost is real, though. The HLSL code in glslang/HLSL/ is frozen, but the shared infrastructure is not. The common code paths in shared code will continue to be touched by upstream GLSL changes. A fork maintainer would need to review each such change for its HLSL impact. That review cost does not go away with a fork.

I would not object to a community-maintained fork being publicized as a separate project, and I'm happy to help document the coupling between the HLSL and GLSL paths.

Does this sound reasonable?

Thanks. Diego.

[TheMostDiligent]

OK, there is some time - we will think what's the best way to move forward.