TLS: python×python interop fails with TLSV1_ALERT_UNKNOWN_CA (CERT_OPTIONAL PKIX vs libp2p self-signed certs)

[acul71]

Summary

python-v0.x x python-v0.x perf tests in libp2p/unified-testing fail on all TLS stacks with:

TLS read: TLS alert from peer: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca
SwarmDialAllFailedError: unable to connect

Noise stacks pass; only TLS fails.

Reproduction

1. In unified-testing perf/, use local py-libp2p at 51a50c80 (main) or any commit including fix(tls): reject inbound connections that present no client certificate #1341.
2. Run:

   ./run.sh --impl-select python-v0.x \
     --baseline-ignore "~baselines" \
     --force-image-rebuild \
     --timeout 900 \
     --yes

3. All 4 TLS stacks fail: (tcp|ws) x (tls) x (yamux|mplex).

Baseline repro: unified-testing perf pass perf-08c8be91-132754-07-07-2026 at py-libp2p 51a50c80 — 4/8 pass (noise only), 4/8 fail (TLS only).

Root cause

After #1341, inbound TLS uses ssl.CERT_OPTIONAL in libp2p/security/tls/transport.py (~line 146). This requests a client certificate and triggers OpenSSL PKIX verification. libp2p peers use self-signed identity certificates verified via the libp2p X.509 extension (libp2p TLS spec), not a CA chain.

When the dialer presents its self-signed libp2p cert, the listener's OpenSSL stack rejects it with TLSV1_ALERT_UNKNOWN_CA before post-handshake libp2p extension verification runs.

Why unit tests pass

tests/core/security/tls/test_tls.py and tests/utils/factories.py call trust_peer_cert_pem() before handshake — pre-loading peer certs as trusted CAs. Real interop (perf harness) does not and should not need this.

Expected fix

Align with Go/rustls libp2p TLS:

• Disable PKIX/CA verification at the OpenSSL layer for peer self-signed certs
• Enforce mutual auth via libp2p extension verification in secure_inbound / secure_outbound
• Preserve fix(tls): reject inbound connections that present no client certificate #1341 behavior: reject inbound connections with no client certificate (do not revert to placeholder identity)

Related

• TLS: inbound TLS handshake accepts connections with no client certificate (authentication bypass) #1340 (inbound auth bypass — fixed by fix(tls): reject inbound connections that present no client certificate #1341)
• fix(tls): reject inbound connections that present no client certificate #1341 (require client cert — introduced PKIX failure for self-signed interop)
• Harden Python perf interop: mplex teardown, yamux throughput, and perf_test.py reliability #1344 (perf hardening)
• unified-testing PR connecting open_stream to stream constructor #37

Environment

• py-libp2p: 51a50c803e46a5247b71d368981f67e3db9eaf59
• Python 3.13 (perf Docker image)
• Platform: Linux x86_64