openssl compat errors and mldsa oid fix

[stenslae]

Description

• Fix 1: verifyX509orX509REQ(): add ML-DSA case

verifyX509orX509REQ() was missing a case for ML-DSA public keys, so they returned errors. Added an mldsaOID field that is from the cert's OID in wolfSSL_X509_get_pubkey(). With pkeys with no backing cert, the OID is cached and used for level recognition.

Also fixed the wc_MlDsaKey_PublicKeyDecode() so that auto-detect is checked before the draft mode.

• Fix 2: return 0 when error queue not compiled in

When WOLFSSL_HAVE_ERROR_QUEUE is not defined, all three functions returned (unsigned long)(0 - NOT_COMPILED_IN) (= 174), a non-zero value. This is non-OpenSSL-compatible, so changed all three #else branches to return 0.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10785

Scan targets checked: wolfssl-bugs, wolfssl-src

No new issues found in the changed files. ✅

[github-actions]

MemBrowse Memory Report

linuxkm-standard

• Data: __patchable_function_entries +8 B (+0.0%, 48,248 B)
  No memory changes detected for:
• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-openssl-compat
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-pq
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m4-tls13
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-pq
• gcc-arm-cortex-m7-tls13
• linuxkm-pie
• stm32-sim-stm32h753

[Frauschi]

🐺 Skoll Code Review

Overall recommendation: REQUEST_CHANGES
Findings: 1 total — 1 posted, 0 skipped

Posted findings

• [Medium] ML-DSA verify uses subject key OID instead of the verifying key's OID for parameter-set selection — src/x509.c:8959

Review generated by Skoll via Claude/Codex

[dgarske]

Some of the CI is failing for PQC. Please check. Thanks @stenslae

[dgarske]

Still seeing CI failures "make check failed for: mldsa-no-asn1-fips204-draft-opensslextra (1 config(s) aborted by fail-fast)"

[Frauschi]

Consider folding the new ML-DSA fixtures into certs/renewcerts.sh

renewcerts.sh already has an ML-DSA block, but it only regenerates the per-level mldsa<N>-* certs/keys and ecc-leaf-mldsa44.pem. The cross-level chain this PR adds (mldsa87-ca-*, mldsa65-leaf87ca-*) is documented only as manual commands in README.txt.

Worth wiring the chain into the script: it's the regression fixture for the OID-selection fix here, and since it isn't scripted, the next renewcerts.sh run regenerates the base certs but leaves the CA/leaf chain untouched — so they drift out of sync. It's pure OpenSSL 3.5+ and fits right after the ecc-leaf-mldsa44 section.

I'd leave _oqskeypair.der (needs liboqs) and the raw key-shape vectors as documented-only. Not blocking.

[Frauschi]

LGTM