What Benchmarking 832 Repositories Across 12 Languages Taught Us About Post-Quantum Readiness and Structural Bug Classes

Synthetic micro-benchmarks rarely fail the way production repositories do. Real engineering estates mix generated protobufs, vendored C headers, stale TLS configurations, and half-finished migrations written by both humans and AI coding assistants. To stress-test CipherMap's parser resilience, detection rules, and verification gates under realistic repository conditions, we ran an offline empirical study across 832 real-world open-source repositories spanning all 12 supported languages (Go, Rust, Python, TypeScript/JavaScript, Java, C/C++, C#, Ruby, PHP, Kotlin, and Swift), compiled .jar/.whl archives, and IaC manifests.

Across the entire 832-repository corpus—executed locally with --offline --no-telemetry—CipherMap recorded 0 scanner panics, sustained typical local scan times between 37 ms and 232 ms on standard multi-service repositories, and wrote 0 broken edits during candidate verification runs. Four engineering lessons stood out.

1. Pre-FIPS Draft Drift (kyber768 & dilithium3) Is Already in the Wild

Security teams often assume their repositories fall neatly into “classical RSA/ECC” or “NIST post-quantum.” Our benchmark showed a third category growing rapidly: early-adopter cryptography and LLM-generated patches importing pre-standardization NIST Round 3 identifiers such as kyber768, kyber512, and dilithium3.

Finalized NIST FIPS 203 (ML-KEM) and FIPS 204 (ML-DSA) introduced domain-separation, encapsulation-key hashing, and wire-encoding changes that make Round 3 drafts non-interoperable with FIPS-conformant peers. CipherMap's PQC-PRE-001..005 rules separate pre-FIPS draft drift from classical RSA/ECDSA debt so migration teams do not mistake draft-era libraries for completed FIPS 203/204 readiness.

2. Cryptographic Parsers Disproportionately Co-Locate Unbounded Stream & Path Sinks

Where repositories parse certificates, JWS/JWT envelopes, signed release bundles, or encrypted archives, two structural bug classes appeared repeatedly alongside classical cryptographic calls:

• Uncapped archive and HTTP stream reads (IMM-BOUND-001): Call sites reading untrusted HTTP bodies, PEM bundles, or compressed .zip/.tar.gz entries into memory without an explicit byte ceiling (for example, bare io.ReadAll in Go or unbounded stream buffers in Python, Node.js, and Java) before signature verification completes.
• Unjailed archive and artifact path joins (IMM-PATH-001): Bundle extractors and keystore loaders joining archive entry names directly onto destination directories without lexical containment verification (Zip-Slip / traversal sinks) prior to manifest or signature checks.

Discovering these sinks locally with ciphermap immunity scan (Tier B Enterprise Fleet, 0 bytes egress — with built-in ciphermap demo sandbox evaluation) lets AppSec teams review input-handling boundaries in the exact packages scheduled for cryptographic refactoring.

3. Why Lazy T0/T1 Baseline Isolation Prevented Broken Repositories

The most consequential systems finding came from running build and test verifiers across unfamiliar open-source checkouts. In real-world fleets, an unmodified repository (T0) frequently fails to compile out-of-the-box on a clean runner because of missing system C libraries, uninitialized submodules, or environment-specific test fixtures.

A remediation tool that skips baseline checks risks committing broken code; a tool that runs post-patch builds without baseline attribution falsely blames its own diff for pre-existing repository errors.

CipherMap resolves this through ciphermap preflight and lazy T0/T1 differential fault attribution:

# 1. Check host toolchains & workspace readiness locally (0 egress)
ciphermap preflight .

# Outcome taxonomy recorded before any edit can be retained:
# - MISSING_HOST_DEPENDENCY  -> Compiler, SDK, or lockfile missing on host (0 edits written)
# - PRE_EXISTING_REPO_ERROR  -> Baseline T0 fails before patching (0 edits written)
# - CANDIDATE_REGRESSION     -> T0 passed, candidate T1 or self-scan failed (atomic rollback)
# - VERIFIED_CANDIDATE       -> T0 passed, sandboxed T1 passed, differential self-scan clean

Because T0 executes lazily—only when a supported recipe produces an eligible candidate—repositories with no applicable rewrites incur zero compiler overhead, while repositories with broken baselines fail closed with a clear PRE_EXISTING_REPO_ERROR diagnostic instead of corrupted working trees.

4. Methodological Scope & Boundaries

Empirical honesty requires stating what an 832-repository static and sandboxed benchmark does not establish. Pattern and AST matches identify supported source, archive, and configuration signals at a point in time; they do not prove runtime reachability in a deployed environment, discover live HSM key estates, or constitute a compliance certification. Ambiguous inter-procedural data flows and wire-protocol changes are intentionally declined (declined_protocol_guardrail) for human architectural review.