riscv64: make RVV TRSM (_rvv_v1) kernels VLEN-agnostic#5928
Conversation
The trsm_kernel_{LN,LT,RN,RT}_rvv_v1.c kernels tiled packed-A along M by
the runtime VSETVL_MAX, but the GEMM packing routine they interoperate
with packs by the compile-time GEMM_UNROLL_M. When VSETVL_MAX differs
from GEMM_UNROLL_M the diagonal-block stride assumed by solve() no longer
matches the packed layout and the solve reads/writes the wrong blocks.
On x280 the two coincide (both 16) so the existing target is unaffected,
but the kernels are not actually VLEN-agnostic: enabling them on a target
where VSETVL_MAX != GEMM_UNROLL_M (e.g. RISCV64_ZVL128B, unroll 8)
corrupts the result.
Replace the outer control flow so packed-A is tiled by GEMM_UNROLL_M with
a power-of-2 M remainder per N-block, matching the generic
kernel/generic/trsm_kernel_{LN,LT,RN,RT}.c drivers. The RVV-vectorized
solve() is retained unchanged.
Validated on SpaceMiT X60 (VLEN=256): S/D/C/Z TRSM over a full
Side/Uplo/Trans/Diag grid, m,n in {1,2,4,7,8,9,10,12,15,23}.
- ZVL128B (unroll 8): 8000/8000 cases pass; the original kernels
fail 408/1600 DTRSM (worst residual 0.89).
✅ Unroll-16 (ZVL256B) confirmation — VLEN-agnostic provenFollow-up to the note in the PR description. I built the same fix as
Combined with the ZVL128B (unroll 8) results in the description, this confirms the kernels are correct at both Marking ready for review. Validated on SpaceMiT X60 hardware, GCC 14.3.0. |
Performance check — no regression (SpaceMiT X60, RVV 1.0, VLEN=256)I benchmarked the fixed kernels against the original Δ = fixed vs original, MFlops:
Important caveat: on Correctness: |
…EN-fix sweep + perf Rename dgemm/ -> OpenBLAS/ (it verifies the whole OpenBLAS RISC-V backend, not just DGEMM). Add verify_ctrsm.c, the full-parameter TRSM correctness sweep used to validate the VLEN-agnostic _rvv_v1 fix (OpenMathLib/OpenBLAS#5928): 408/1600 -> 8000/0 on ZVL128B, also 8000/0 on ZVL256B. README gains a TRSM section (root cause, correctness table, fix summary, no-regression perf table with broken-baseline caveat). Fix ../dgemm links in root/numpy/hpl READMEs.
1 participant
Make the RVV
_rvv_v1TRSM kernels VLEN-agnosticProblem
The
kernel/riscv64/trsm_kernel_{LN,LT,RN,RT}_rvv_v1.ckernels tile packed-A alongMby the runtimeVSETVL_MAX, but the GEMM packing routine they interoperate with packs by the compile-timeGEMM_UNROLL_M. WhenVSETVL_MAX != GEMM_UNROLL_M, the diagonal-block stride assumed bysolve()no longer matches the packed layout, so the solve reads/writes the wrong blocks and the result is corrupted.On x280 the two happen to coincide (both 16), so the currently shipping target is unaffected — but the kernels are not actually VLEN-agnostic. Enabling them on any target where
VSETVL_MAX != GEMM_UNROLL_M(e.g.RISCV64_ZVL128B, which packs at unroll 8) produces wrong results. This is exactly what surfaces in #5830, which enables these kernels for ZVL128B.Fix
Replace only the outer control flow so packed-A is tiled by
GEMM_UNROLL_M(with a power-of-2Mremainder perN-block), matching the generickernel/generic/trsm_kernel_{LN,LT,RN,RT}.cdrivers. The RVV-vectorizedsolve()is retained unchanged. This makes the kernels correct on both unroll-8 (ZVL128B) and unroll-16 (x280) targets.Validation (SpaceMiT X60, VLEN=256, GCC 14.3.0)
Residual test
‖op(A)·X − B‖/‖X·op(A) − B‖overm,n ∈ {1,2,4,7,8,9,10,12,15,23}× every Side/Uplo/Trans/Diag.Built as
RISCV64_ZVL128B(unroll 8), TRSM wired to_rvv_v1:Regression contrast, same harness, same target:
_rvv_v1(VSETVL_MAXtiling)GEMM_UNROLL_Mtiling)An unroll-16 (ZVL256B-params) build is being validated on the same hardware; I'll post the result as a follow-up. Since x280 already uses these kernels at unroll 16, the change is expected to be a no-op there (16 == 16) while hardening against future VLEN/UNROLL divergence.
Notes
_rvv_v1for ZVL128B). The fix here is what makes that enablement correct; happy to coordinate.Validated on SpaceMiT X60 hardware.