CAIN-42 MCPGateCAIN-42 four-server cluster: loss, jitter, duplication, reordering
Degraded-network test on the live CAIN-42 cluster cain-mr-02 (4 replicas on 4 servers in 4 regions): packet loss, delay with jitter, duplication and reordering on every replica's traffic at once. This page loads every quorum certificate all four replicas hold afterwards, and your browser re-checks them.
What happened
Impairment on all four replicas' mesh traffic: delay 120ms 40ms distribution normal loss 10% duplicate 5% reorder 10% 50% (all 4 hosts, cain-mr-02 overlay traffic only).
| phase | length | writes committed | rate | p50 | p95 |
|---|---|---|---|---|---|
| baseline | 45 s | 79/79 | 1.76/s | 549.3 ms | 902.8 ms |
| impaired | 240 s | 39/61 | 0.16/s | 3539.7 ms | 7173.9 ms |
| recovered | 45 s | 91/91 | 2.02/s | 520.7 ms | 644.0 ms |
Safety held: the identical decision chains below, on all four replicas, show there was no fork. Liveness degraded sharply: under the impairment the commit rate fell by about an order of magnitude and some writes did not commit within the client's 30 s timeout; it recovered fully once the impairment ended (all four agreed 0.4 s after). That is a measured limit, not a pass on performance.
certificates re-collected at one stable height: the first collection straddled the 03:38 proof commit (seq 341), so one replica's status described a later height than its certificates; the replicas never diverged.
Method: tc prio qdisc + u32 filter (destination = the cluster overlay subnet) -> netem band, removed by a timer on each host.
Verify (about 5 seconds)
What is checked
- The membership configuration hash is recomputed from the 4 member ids and Ed25519 public keys. Every node and every certificate must carry it.
- For every sequence on every node, the COMMIT_QC and the PREPARE_QC. Each vote must be an Ed25519 signature by a distinct member over SHA-256 of the canonical signed message. It must have the right type (a COMMIT vote never counts as a PREPARE vote) and match this cluster, epoch, view, sequence and digest. Each certificate needs at least 3 distinct signers. The leader's proposal must be signed by the primary of that view, and its digest must bind the proposed operation.
- The certificate hash and signature-bundle hash are recomputed from the content.
- Evolution 3 fast path: a
FAST_COMMIT_QC(a decision taken without the COMMIT round) is accepted only if the published membership declares the fast path and all four members signed it. Three of four is never enough for a fast commit. - The decision chain is folded from genesis:
decision_hash(seq) = H(cluster, epoch, seq, digest, parent). It must be contiguous and identical on all four nodes, and the nodes must end with the same application-state hash. - View-change quorum certificates, and one consensus-to-enforcement AuthorizationCertificate per node.
- Negative controls: a certificate is tampered with in seven ways, and every tampered copy must be rejected.
The same checks, without a browser: curl -so verify_pbft_qc_bundle.py verify_pbft_qc_bundle.py.txt && python3 verify_pbft_qc_bundle.py PBFT_QC_BUNDLE.json (needs pip install cryptography, no CAIN code).