Bouncy Castle C# Vulnerable to Subgroup Confinement Attack Breaking Diffie-Hellman Key Agreement (CVE-2026-63569)

Executive Summary: Cryptographic Failure in Core .NET Security Library

The Legion of the Bouncy Castle cryptography project has published a critical security advisory (GHSA-m2m9-m35w-rfxg) detailing a severe mathematical vulnerability in its widely used C# implementation (bc-csharp). Cataloged as CVE-2026-63569, the flaw impacts applications calling DHAgreement.CalculateAgreement—the engine implementing the Matsumoto-Takashima-Imai (MTI/A0) two-pass authenticated Diffie-Hellman key agreement protocol.

The vulnerability stems from improper input validation on incoming ephemeral public values. Because the engine fails to perform strict range and subgroup-membership checks, an on-path attacker or malicious remote peer can transmit a crafted out-of-range or small-order group element. When the local party computes the key agreement, the untrusted element is raised directly to the local static private key. This enables the attacker to force the local party into calculating a predictable session key and progressively learn the local static private key modulo small factors of p - 1, culminating in complete private key recovery via the Pohlig-Hellman algorithm.

Mathematical Mechanics of the Subgroup Confinement Attack

In a standard Diffie-Hellman key exchange over a finite multiplicative group (mathbb{Z}_p^*), parties agree on a large prime (p) and a generator (g). In the MTI/A0 two-pass protocol, both static and ephemeral key pairs are combined to provide implicit mutual key authentication:

• Alice possesses static private key (a) and static public key (A = g^a pmod p), alongside ephemeral key pair ((x, X = g^x pmod p)).
• Bob possesses static private key (b) and static public key (B = g^b pmod p), alongside ephemeral key pair ((y, Y = g^y pmod p)).
• The agreed shared secret (K) is derived as (K = A^y cdot Y^a = g^{ay + ya} pmod p) (or symmetric protocol variant).

According to cryptographic standards (including NIST SP 800-56A Rev. 3 and IEEE P1363), any implementation receiving a public value (Y) from an untrusted party MUST strictly verify:

1. Range Verification: (2 le Y le p - 2).
2. Subgroup Membership Verification: (Y^q equiv 1 pmod p), where (q) is the large prime subgroup order dividing (p - 1).

In affected versions of bc-csharp prior to 2.7.0, DHAgreement.CalculateAgreement applied subgroup checks to static public keys, but omitted them entirely for incoming ephemeral values.

Exploit Sequence & Private Key Recovery Flow

An attacker selects an element ( ilde{Y}) belonging to a small subgroup of (mathbb{Z}_p^*) of small order (r), where (r) divides (p - 1). When the vulnerable library calculates:

// Vulnerable implementation in DHAgreement.cs (pre-2.7.0):
public BigInteger CalculateAgreement(DHPublicKeyParameters pub, BigInteger message)
{
    // 'message' represents the peer's incoming ephemeral value.
    // Notice: No validation that 2 <= message <= p-2
    // Notice: No check that message^q mod p == 1
    BigInteger result = message.ModPow(key.X, dhParams.P); // key.X is the static private key 'a'
    return result;
}

Because ( ilde{Y}) has order (r), the computed value ( ilde{K} = ilde{Y}^a pmod p) can only take one of (r) possible distinct values. The adversary tests candidate values for (a pmod r). By repeating this process across multiple handshake sessions using small factors (r_1, r_2, dots, r_k) of (p - 1), the adversary applies the Chinese Remainder Theorem (CRT) to reconstruct the victim's static private key:

# Mathematical attack flow:
a = a1 (mod r1)
a = a2 (mod r2)
...
a = ak (mod rk)
==> a = CRT(a1, a2, ..., ak) mod (r1 * r2 * ... * rk)

If the prime (p) was generated without safe-prime criteria (i.e. (p - 1) is smooth and contains many small factors), the entire static private key (a) is recovered within a matter of seconds.

Impact on Enterprise Applications

Defensive Remediation Guidelines

1. Upgrade to Bouncy Castle C# 2.7.0: Immediately update NuGet package references in .NET projects from BouncyCastle.Cryptography or Portable.BouncyCastle to release 2.7.0 or newer.
2. Enforce Safe-Prime Group Generation: When using finite-field Diffie-Hellman, only use safe primes (p = 2q + 1) where (q) is also prime (such as RFC 3526 / RFC 7919 predefined groups). Avoid generating custom primes with smooth (p - 1).
3. Transition to Modern Elliptic Curve Cryptography: Migrate legacy finite-field Diffie-Hellman workflows to X25519 (Curve25519) or ECDH with NIST P-384, which offer mathematically robust small-subgroup resistance by design.