Add AArch64 SIMD: ARMv8 Crypto Extensions AES, PMULL GHASH/POLYVAL, and NEON ChaCha/Salsa/Poly1305#154
Merged
Conversation
…rch64) First round of AArch64 SIMD support, mirroring the x86 AES stack: - TAesEngineArm (ClpAesEngineArm/ClpIAesEngineArm): hardware AES engine gated on CRYPTOLIB_AARCH64_ASM, dispatched via TCpuFeatures.Arm.HasAES and the TAesSimd facade; same IBulkBlockCipher 8/4/1 ladder and TryGetEnc/DecKeysPtr contract as TAesEngineX86 (the inverse schedule layout is identical, so fused kernels consume it unchanged). - Fused kernels: 8-wide CTR keystream+XOR (ClpAesCryptoExtCtrKernel, same 2^32 wrap-splitting contract as the x86 kernel; counters staged raw since aese consumes K_0 first) and CBC (ClpAesCryptoExtCbcKernel: serial encrypt, 8-wide fused decrypt with in-place-safe chain XOR), registered through the existing kernel registry - no mode changes. - Kernels are .long-encoded instruction streams (FPC 3.2.2's AArch64 assembler lacks the vector/crypto mnemonics), verified opcode-for- opcode against GNU binutils for every width/key-size/direction variant. Key expansion is constant-time in vector registers (aese-with-zero SubWord, ext RotWord, movi rcon, doubling prefix fold); all kernels stay off v8..v15, so no save frames are needed. - Shared AArch64 prologues ClpSimdProc1..6Begin_aarch64.inc plus (currently unconsumed) ClpSimdNonVolatileSave/Restore_aarch64.inc. - x86 alignment: the CBC kernels (EncryptSerial + DecryptWide, both arches) now embed their round chains, matching the AArch64 layout - proven instruction-identical to the previous nested-include form for all 12 arch/variant/key-size streams - and the ROUNDS_ONLY selector is removed along with the inert CRYPTOLIB_AESNI_DECRYPT defines in ClpAesNiCbcKernel.
Second round of AArch64 SIMD support: the GCM/GHASH family, mirroring the x86 (PCLMULQDQ) stack and byte-compatible with its state, limb and table formats: - TGhashArmBackend (ClpGhashArmBackend): the full 9-entry GHASH surface behind the TGhashSimd facade - carryless multiply/multiply-ext, limb fold+reduce, limb XOR, block byte-reverse, and the monolithic 4-/8-way batch GHASH kernels looping over ABatchCount batches in-asm. The H-power table convention is unchanged (H^N..H^1, pre-multiplied by x in the reflected representation), so the batch product reduces with two carry-less multiplies by 0xC2000000_00000000; the running state stays reflected across batches. rev64 + half-swap replaces the x86 shuffle mask, so no byte-reverse constant is needed; the GHASH batch wrappers drop the mask parameter accordingly. The scalar partial reduction is identical to the x86 backend's. GCM_GHASH_RAW_INPUT is supported for the future GCM-SIV POLYVAL port. - TAesCryptoExtGcmKernel (ClpAesCryptoExtGcmKernel): fused AES CTR keystream + 8-way PMULL GHASH kernel behind IGcmKernel, with the AES and carry-less-multiply instruction streams interleaved so both pipes run in parallel (GHASH lags the keystream by one batch; the mode primes and drains). Context layout is identical to the x86_64 kernel (PMask kept for parity, unused). Registered via the kernel registry - no mode changes. First consumer of the d8..d15 save/restore includes (the GHASH accumulators live in v8..v15 alongside 8 AES lanes and the resident round-key schedule). - Facades: ClpGhashSimd forwards all nine entries to the ARM backend under CRYPTOLIB_AARCH64_ASM; ClpAesFusedAeadSimd.CpuSupports gains the matching arm (TAesCryptoExtFusedArmBackend). - Kernels are .long-encoded instruction streams verified opcode-for- opcode against GNU binutils and validated against a from-spec GHASH reference (field multiply, DivideP table transform, batch semantics, and the fused pipeline incl. counter write-back) for all key sizes and both directions.
Third round of AArch64 SIMD support: AES-GCM-SIV, completing the GHASH-family ports on ARM: - TGcmSivArmBackend (ClpGcmSivArmBackend): PMULL 8-block POLYVAL Horner batch behind the TGcmSivSimd facade. The kernel is the raw-input variant of the existing AArch64 GHASH batch body (POLYVAL supplies blocks already in field form, so the per-block byte-reverse is skipped; the running-state load and final store are still byte-reversed) - no new assembly. No shuffle mask is needed on AArch64, so the wrapper drops the mask parameter; the facade's PMask argument is accepted for parity and unused. - TPmullGcmSivKernel (ClpPmullGcmSivKernel): ARM implementation of IGcmSivKernel mirroring the PCLMULQDQ kernel (pure POLYVAL, factory ignores the cipher identity and only requires the caller's pre-computed H-power table). Registered via the kernel registry - no mode changes. - ClpGcmSivSimd forwards IsSupported / ProcessPolyvalBatch to the ARM backend under CRYPTOLIB_AARCH64_ASM. - TPclmulGcmSivKernelFactory.TryCreate body is now gated on CRYPTOLIB_X86_SIMD: it previously relied on TGcmSivSimd.IsSupported being implicitly x86-only, which no longer holds once the facade dispatches per-arch. No behavior change on x86 (the gate compiles away there).
Port the 8-wide fused OCB body to ARMv8 Crypto Extensions, matching the x86_64 kernel op-for-op: two 8-block iterations per pass, the next iteration's offset ladder staged between AES round groups into ping-pong stack delta banks, ntz derived branch-free from the 64-bit block counter via rbit+clz, and the offset/checksum accumulators in v8/v9 bracketed by the shared d8..d15 save/restore frames. No K_0 pre-whitening is needed on AArch64 (aese consumes the first schedule slot), so lanes are whitened with the stashed deltas alone. Context layout is identical to the x86_64 kernel's. The kernel is emitted by a new "ocb" group in gen-aes-cryptoext-simd.py (new rbit/clz/lsr/sp-arithmetic encoders); all six variant streams are binutils-oracle verified and emulator-validated against a from-spec OCB reference, including start counts past 2^32, a detached block-0 source and in-place operation. TAesCryptoExtOcbKernel registers via the kernel registry with the same 16-block minimum stride as the x86_64 kernel. The AES-NI OCB factory body is now compile-gated on CRYPTOLIB_X86_SIMD so it cannot misfire on ARM through the shared fused-AEAD facade.
Port the fused CTR+CBC-MAC (CCM) and CTR+OMAC (EAX) bodies to ARMv8 Crypto Extensions. CCM-encrypt/EAX-decrypt and CCM-decrypt/EAX-encrypt share their two instruction shapes (MAC folds input vs previous output). Round keys stay resident in v16.. and rev64+ext replaces the shuffle mask; both directions consume the forward schedule with the key pointer re-resolved per call. Context layouts match the x86_64 kernels'.
Port the GF(2)[x] BinPoly kernel family to ARMv8 PMULL. The fixed-size SMALL cases (1..10 limbs)
Port the Salsa20 kernel pair to NEON, matching the x86 SSE2 contracts: the diagonal lane layout is gathered with ldr+ins and scattered with umov+str (fused I/O xors through a GPR), rotations use shl+ushr+orr, and the per-block counter advance over state words 8/9 is a single 64-bit add in place of the x86 carry branch. Originals stay register-resident, so no stack copy or save frames are needed. TSalsaArmBackend serves both tiers through new AArch64 facade arms.
The fixed-size multiplies for 9 and 10 limbs accumulated output slots in v0..v9, and the looped kernels' 2x2 Karatsuba tile produced its two slots in v8/v9 - clobbering callee-saved registers without a save frame. Remap those to the free caller-saved v16/v17 so every BinPoly path stays frame-free.
Benchmark Poly1305 and GHASH through the public IMac API: Poly1305 directly, GHASH as GMAC-AES-128, both taking whatever path the runtime CPU dispatch selects - accelerated where supported, scalar fallback otherwise - so the same binary reports meaningful numbers on any machine. Each timed iteration is the full per-message operation (Init, absorb, tag): Poly1305 keys are single-use by contract and GCM rejects nonce reuse, so the GMAC runner advances its nonce per message.
Port the 2-lane radix-2^26 Poly1305 bulk kernel to NEON, matching the x86 SSE2 chain's contract: the same r/r^2 power table, 5x5 schoolbook multiply mod 2^130-5 with the 5*r wraparound, serial lazy reduction, and tail multiply by [r^2, r^1] with a 2-lane horizontal fold. Multiply inputs live as 32-bit lanes so each of the 25 products is one umull/umlal into a 64-bit accumulator; the reduction runs on the 64-bit accumulators and narrows to the multiply form only at the end of the chain, since the 5*carry fold can exceed 32 bits. The bulk table rows stay register-resident, no aligned staging copy is needed, and no callee-saved vector registers are touched.
CCM's length field gets q = 15 - nonceLen bytes, so the GCM-style 12-byte nonce caps messages at 2^24 - 1 bytes and a 16 MB benchmark payload fails with "CCM packet too large for choice of q". An 11-byte nonce (q = 4) lifts the ceiling to 2^32 - 1 with no effect on measured throughput.
Horner body (per the RFC 8452 identity); the AArch64 backend carried the same delegation inline instead. Move it into GcmSiv/PolyvalHornerEight_aarch64.inc with the matching header, so both backends read identically and Include/Simd/GcmSiv covers every architecture. No code change - the delegated instruction stream is untouched.
- ByteXor: NEON variable-length XOR kernel (16-byte chunks + byte tail, alias-safe) behind TByteXorArmBackend, wired into the ByteXor facade; GCM-SIV tag/keystream XORs and CFB decrypt now take the SIMD path on AArch64 instead of the scalar fallback. - Poly1305: replace the 2-way NEON bulk kernel with a 4-way port of the x86 4-way structure - four radix-2^26 Horner chains as two vector halves, a 320-byte r^1..r^4 power table, two-interleaved-chain lazy reduction, and a [r^4, r^3, r^2, r^1] tail multiply with 4-lane fold. The bulk path now consumes 4 blocks per iteration (minimum batch 4), matching the x86 kernel's chain-depth parallelism.
Salsa20 previously topped out at the 2-block kernels on every
architecture. This adds the same wide streaming tiers ChaCha has:
- AArch64: 4-way vertical NEON kernel (one block per lane, all four
diagonal constants re-broadcast so four rotate temps stay free;
shl+sri rotations; 64-bit words-8/9 counter write-back per group).
- x86_64/i386: 4-way SSE2 and 8-way AVX2 vertical kernels. All
rotation amounts (7/9/13/18) are shift+or - none is byte-aligned,
so there is no pshufb tier. On x86_64 the state stays register-
resident; the quarter-round word sets partition so that parking
{2,3,6,7} or {8,9,12,13} alternately costs two 4-store/4-load
swaps per double round, and the output tail drains the resident
word quads first so no spill area is needed.
- Engine: 8 -> 4 -> 2 -> 1 bulk ladder with a wide-group clamp so
the low counter word cannot wrap inside a wide span (the kernels
broadcast the high counter word once per call). XSalsa20 inherits
the ladder unchanged.
…tier The x86 stream-cipher backends used case SelectSlot([...]) uniformly, including for methods with only one tier, while the feature-gate backends (Ghash, ByteXor, GcmSiv, BinPoly, AES-fused) and every ARM backend used if HasX. SelectSlot ranks a linear SIMD-level ladder, so it cannot express the orthogonal feature bits (PCLMUL, AESNI, PMULL, AES) those backends gate on - meaning "case everywhere" was never possible. Settle on the reverse rule instead: case SelectSlot only where a method genuinely chooses among >= 2 ladder levels; if HasX() otherwise. - ChaCha X86: TryCore / TryProcessBlocks2 / TryProcessBlocks8 -> if; TryProcessBlocks4 keeps its case (real SSSE3 + SSE2 choice). - Salsa X86: all four Try* -> if; drop the now-unused ClpSimdLevels. - Poly1305 X86 keeps its AVX2 + SSE2 case; all other backends already matched the rule. Also parenthesize every CPU-feature call (HasSSE2 -> HasSSE2(), etc.) across all x86 backends so they read as the method calls they are, matching the ARM backends' HasNEON()/HasPMULL() convention.
The NEON ChaCha quarter-round rotated with ushr+shl+orr (3 ops) for rot12/rot8/rot7, making the keystream slower on ARM than Salsa's - the inverse of x86, where ChaCha's byte-permute rotations make it the faster cipher. Fold the preceding xor into the rotate scratch and rotate with shl + sri instead: eor t,v,x; shl v,t,n; sri v,t,32-n lands the rotated word back in its home register with no extra move and no register renaming. rot16 stays a single rev32. This drops the quarter-round from 18 to 15 instructions - already cheaper than Salsa's 16 (ChaCha gets rot16 free via rev32), so ARM ChaCha keystream should match or beat Salsa. Applied to the 4-way bulk kernel and the horizontal core / 2-block kernels.
TIntrinsicsVector.IsPacked checked that SizeOf(packed array[0..N] of Byte) equals N+1, which is guaranteed by the language on every target (Byte is 1 byte; byte arrays are never padded). So it always returned True, and the `and TIntrinsicsVector.IsPacked` tacked onto the x86 GHASH / GCM-SIV / BinPoly / AES-NI-fused feature predicates was dead weight. The unit's three vector-byte types were used nowhere but inside that check, and the AArch64 PMULL backends already run the same 16-byte SIMD kernels with no such guard.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR brings CryptoLib4Pascal to feature parity with the existing x86 SIMD stack on AArch64, wiring accelerated paths through the same cipher-kernel and backend/facade architecture.
ARMv8 Crypto Extensions AES engine
TAesEngineArm(IAesHardwareEngine) with AES-NI-equivalent bulkProcessBlocks(count)and fused CTR/CBC kernelsClpAesCryptoExtCtrKernel,ClpAesCryptoExtCbcKernel,ClpAesCryptoExtGcmKernel,ClpAesCryptoExtOcbKernel,ClpAesCryptoExtCcmKernel,ClpAesCryptoExtEaxKernelFused AES AEAD kernels (AArch64)
ClpPmullGcmSivKernel; dedicated AArch64 POLYVAL shell includePMULL binary-polynomial math
TTestBinPolyArm(PMULL backend parity against scalar reference)Stream ciphers + MACs (NEON)
rev32+shl/srirotationsByteXorbackend (shared XOR primitive for fused paths)SIMD infrastructure
ClpArmSimdFeatures, hwcap provider) for NEON, AES, PMULL, etc.ClpChaChaArmBackend,ClpSalsaArmBackend,ClpPoly1305ArmBackend,ClpGhashArmBackend,ClpGcmSivArmBackend,ClpBinPolyArmBackend,ClpByteXorArmBackend,ClpAesCryptoExtFusedArmBackendiffor single-tier,caseonly for multi-tierClpIntrinsicsVectorguard.inchelpers; rename x86 ChaCha/Salsa block includes for consistent 2/4/8-way namingBenchmark