Merge pull request #2300 from folkertdev/extern-custom

Add `extern "custom"`

Author: traviscross

src/attributes/codegen.md

@@ -110,6 +110,17 @@ The `cold` attribute may only be applied to functions with [bodies] --- [closure
 
 <!-- TODO: rustc currently seems to allow cold on a trait function without a body, but it appears to be ignored. I think that may be a bug, and it should at least warn if not reject (like inline does). -->
 
+r[attributes.codegen.cold.extern-custom]
+The `cold` attribute may not be applied to an [`extern "custom"` function].
+
+```rust,compile_fail
+#[cold] // ERROR: Not allowed.
+#[unsafe(naked)]
+unsafe extern "custom" fn f() {
+    core::arch::naked_asm!("ret")
+}
+```
+
 r[attributes.codegen.cold.duplicates]
 Only the first use of `cold` on a function has effect.
 
@@ -144,6 +155,9 @@ The *`naked` [attribute]* prevents the compiler from emitting a function prologu
 > # }
 > ```
 
+> [!NOTE]
+> The assembly code of a naked function often does not follow the calling convention of any ABI known to the compiler. Such a function should be declared as an [`extern "custom"` function][items.fn.extern.custom].
+
 r[attributes.codegen.naked.syntax]
 The `naked` attribute uses the [MetaWord] syntax.
 
@@ -900,6 +914,7 @@ If the address of the function is taken as a function pointer, the low bit of th
 [`-C target-cpu`]: ../../rustc/codegen-options/index.html#target-cpu
 [`-C target-feature`]: ../../rustc/codegen-options/index.html#target-feature
 [`export_name`]: abi.export_name
+[`extern "custom"` function]: items.fn.extern.custom
 [`inline` attribute]: attributes.codegen.inline
 [`is_aarch64_feature_detected`]: ../../std/arch/macro.is_aarch64_feature_detected.html
 [`is_x86_feature_detected`]: ../../std/arch/macro.is_x86_feature_detected.html

src/items/external-blocks.md

@@ -48,7 +48,7 @@ r[items.extern.fn.param-patterns]
 Patterns are not allowed in parameters, only [IDENTIFIER] or `_` may be used.
 
 r[items.extern.fn.qualifiers]
-The `safe` and `unsafe` function qualifiers are allowed, but other function qualifiers (e.g. `const`, `async`, `extern`) are not.
+The `safe` and `unsafe` function qualifiers are allowed, but other function qualifiers (e.g. `const`, `async`, `extern`) are not. The `safe` qualifier is rejected in `extern "custom"` blocks.
 
 r[items.extern.fn.foreign-abi]
 Functions within external blocks may be called by Rust code, just like functions defined in Rust. The Rust compiler automatically translates between the Rust ABI and the foreign ABI.
@@ -112,6 +112,9 @@ r[items.extern.abi.system]
 r[items.extern.abi.unwind]
 * `extern "C-unwind"` and `extern "system-unwind"` --- Identical to `"C"` and `"system"`, respectively, but with [different behavior][unwind-behavior] when the callee unwinds (by panicking or throwing a C++ style exception).
 
+r[items.extern.abi.custom]
+* `unsafe extern "custom"` --- A custom ABI that is not known to the compiler.
+
 r[items.extern.abi.platform]
 There are also some platform-specific ABI strings:
 

src/items/functions.md

@@ -262,6 +262,67 @@ With `panic=unwind`, when a `panic` is turned into an abort by a non-unwinding A
 
 For other considerations and limitations regarding unwinding across FFI boundaries, see the [relevant section in the Panic documentation][panic-ffi].
 
+r[items.fn.extern.custom]
+### Extern "custom"
+
+r[items.fn.extern.custom.intro]
+An `extern "custom"` function has an unknown, custom ABI. The only way to call such a function is via [inline assembly].
+
+> [!EXAMPLE]
+> ```rust
+> # #[cfg(target_arch = "x86_64")] {
+> # use core::arch::{asm, naked_asm};
+> #
+> /// Adds 1 to `rax`.
+> ///
+> /// This function uses a custom calling convention: the argument is
+> /// passed in `rax`, the result is returned in `rax`, the flags may
+> /// be clobbered, and all other registers are preserved.
+> #[unsafe(naked)]
+> unsafe extern "custom" fn increment() {
+>     naked_asm!(
+>         "add rax, 1",
+>         "ret",
+>     )
+> }
+>
+> let mut x: u64 = 41;
+> // SAFETY: The inline assembly respects the calling convention of
+> // `increment`: the argument is passed in `rax`, the result is read
+> // from `rax`, and no other registers are affected.
+> unsafe {
+>     asm!(
+>         "call {}",
+>         sym increment,
+>         inout("rax") x,
+>     );
+> }
+> assert_eq!(x, 42);
+> # }
+> ```
+
+r[items.fn.extern.custom.signature]
+An `extern "custom"` function must:
+
+- Be `unsafe`.
+- Not have any parameters.
+- Return the [unit type], with the return type either omitted or written explicitly as `()`.
+
+> [!NOTE]
+> The rule is syntactic. The return type may not be a type alias, even one defined to be the [unit type].
+>
+> ```rust,compile_fail
+> type Unit = ();
+>
+> #[unsafe(naked)]
+> unsafe extern "custom" fn f() -> Unit { // ERROR: Not explicit `()`.
+>     core::arch::naked_asm!("ret")
+> }
+> ```
+
+r[items.fn.extern.custom.naked]
+An `extern "custom"` function definition must be a [naked function].
+
 [forced-unwinding]: https://rust-lang.github.io/rfcs/2945-c-unwind-abi.html#forced-unwinding
 [panic handler]: ../panic.md#the-panic_handler-attribute
 [panic-ffi]: ../panic.md#unwinding-across-ffi-boundaries
@@ -646,6 +707,7 @@ fn foo_oof(#[some_inert_attribute] arg: u8) {
 [testing attributes]: ../attributes/testing.md
 [`cold`]: ../attributes/codegen.md#the-cold-attribute
 [`inline`]: ../attributes/codegen.md#the-inline-attribute
+[naked function]: ../attributes/codegen.md#the-naked-attribute
 [`deprecated`]: ../attributes/diagnostics.md#the-deprecated-attribute
 [`doc`]: ../../rustdoc/the-doc-attribute.html
 [`must_use`]: ../attributes/diagnostics.md#the-must_use-attribute
@@ -664,3 +726,4 @@ fn foo_oof(#[some_inert_attribute] arg: u8) {
 [`VaList<'_>`]: lang-types.va-list
 [`VaList`]: lang-types.va-list
 [zero-sized]: glossary.zst
+[inline assembly]: ../inline-assembly.md

src/types/function-pointer.md

@@ -50,6 +50,9 @@ The `unsafe` qualifier indicates that the type's value is an [unsafe function],
 r[type.fn-pointer.constraint-variadic]
 For the function to be variadic, its `extern` ABI must be one of those listed in [items.extern.variadic.conventions].
 
+r[type.fn-pointer.extern-custom]
+An `extern "custom"` function pointer must follow the rules in [items.fn.extern.custom.signature].
+
 r[type.fn-pointer.attributes]
 ## Attributes on function pointer parameters