CAIN-42 MCPGateCAIN-42 four-server cluster: loss, jitter, duplication, reordering
Degraded-network test on the live CAIN-42 cluster cain-mr-02 (4 replicas on 4 servers in 4 regions): packet loss, delay with jitter, duplication and reordering on every replica's traffic at once. This page loads every quorum certificate all four replicas hold afterwards, and your browser re-checks them.
What happened
Impairment on all four replicas' mesh traffic: delay 120ms 40ms distribution normal loss 10% duplicate 5% reorder 10% 50% (all 4 hosts, cain-mr-02 overlay traffic only).
| phase | length | writes committed | rate | p50 | p95 |
|---|---|---|---|---|---|
| baseline | 45 s | 68/68 | 1.51/s | 596.4 ms | 1103.6 ms |
| impaired | 240 s | 44/62 | 0.18/s | 3880.6 ms | 6212.6 ms |
| recovered | 45 s | 89/89 | 1.98/s | 486.6 ms | 729.4 ms |
Safety held: the identical decision chains below, on all four replicas, show there was no fork. Liveness degraded sharply: under the impairment the commit rate fell by about an order of magnitude and some writes did not commit within the client's 30 s timeout; it recovered fully once the impairment ended (all four agreed 0.4 s after). That is a measured limit, not a pass on performance.
Before and after the engine change
did engine 948b189 (view-change backoff resets on progress, fresh views not accused) improve liveness under this impairment?
Under the impairment: before (521f84c (bundle degraded-network-2026-09-27)) 39/61 committed, 0.16/s, p95 7173.9 ms; after (948b189 (this bundle)) 44/62 committed, 0.18/s, p95 6212.6 ms. View changes during the impairment: before 14 (view 8 -> 22); after at most 6 (view 22 at the 04:08 proof, before a rolling upgrade that itself changes view -> 28).
The view-change storm fell by more than half, but throughput under 10% loss did not improve beyond run-to-run noise (0.16 -> 0.18 commits/s). The storm was not the bottleneck; message delivery under loss is. Safety held in both runs.
Method: tc prio qdisc + u32 filter (destination = the cluster overlay subnet) -> netem band, removed by a timer on each host.
Verify (about 5 seconds)
What is checked
- The membership configuration hash is recomputed from the 4 member ids and Ed25519 public keys. Every node and every certificate must carry it.
- For every sequence on every node, the COMMIT_QC and the PREPARE_QC. Each vote must be an Ed25519 signature by a distinct member over SHA-256 of the canonical signed message. It must have the right type (a COMMIT vote never counts as a PREPARE vote) and match this cluster, epoch, view, sequence and digest. Each certificate needs at least 3 distinct signers. The leader's proposal must be signed by the primary of that view, and its digest must bind the proposed operation.
- The certificate hash and signature-bundle hash are recomputed from the content.
- Evolution 3 fast path: a
FAST_COMMIT_QC(a decision taken without the COMMIT round) is accepted only if the published membership declares the fast path and all four members signed it. Three of four is never enough for a fast commit. - The decision chain is folded from genesis:
decision_hash(seq) = H(cluster, epoch, seq, digest, parent). It must be contiguous and identical on all four nodes, and the nodes must end with the same application-state hash. - View-change quorum certificates, and one consensus-to-enforcement AuthorizationCertificate per node.
- Negative controls: a certificate is tampered with in seven ways, and every tampered copy must be rejected.
The same checks, without a browser: curl -so verify_pbft_qc_bundle.py verify_pbft_qc_bundle.py.txt && python3 verify_pbft_qc_bundle.py PBFT_QC_BUNDLE.json (needs pip install cryptography, no CAIN code).