CAIN-42 MCPGate
Sign inGet started
Menu

CAIN-42 multi-region cluster: fault-injection proof

The live CAIN-42 PBFT cluster cain-mr-01 has 4 replicas on 3 hosts in 3 Vultr regions: Atlanta, Los Angeles ×2 and Miami, connected by a WireGuard mesh (n=4, f=1, quorum 3). During this run we stopped real replicas on their own hosts. Miami down: it kept committing. One Los Angeles replica down: it kept committing. The whole Los Angeles host down (2 of 4): it refused to commit. Atlanta down, including the primary: view change, and it kept committing. Then everything came back. This page loads every quorum certificate all four replicas produced, with the original Ed25519-signed votes, and your browser re-checks them. Nothing here asks you to trust the server. The live cluster page shows the same cluster running now.

Verify (about 5 seconds)

Check the fault schedule against the signatures

A stopped replica cannot sign. For each outage, this checks that none of the decisions taken during it carries a signature from the stopped replicas. It also checks that no request from the refused phase ever entered any decision chain, and that every decision was signed from at least 2 regions.

What is checked

  1. The membership configuration hash is recomputed from the 4 member ids and Ed25519 public keys. Every node and every certificate must carry it.
  2. For every sequence on every node, the COMMIT_QC and the PREPARE_QC. Each vote must be an Ed25519 signature by a distinct member over SHA-256 of the canonical signed message. It must have the right type (a COMMIT vote never counts as a PREPARE vote) and match this cluster, epoch, view, sequence and digest. Each certificate needs at least 3 distinct signers. The leader's proposal must be signed by the primary of that view, and its digest must bind the proposed operation.
  3. The certificate hash and signature-bundle hash are recomputed from the content.
  4. Evolution 3 fast path: a FAST_COMMIT_QC (a decision taken without the COMMIT round) is accepted only if the published membership declares the fast path and all four members signed it. Three of four is never enough for a fast commit.
  5. The decision chain is folded from genesis: decision_hash(seq) = H(cluster, epoch, seq, digest, parent). It must be contiguous and identical on all four nodes, and the nodes must end with the same application-state hash.
  6. View-change quorum certificates, and one consensus-to-enforcement AuthorizationCertificate per node.
  7. Negative controls: a certificate is tampered with in seven ways, and every tampered copy must be rejected.

The same checks, plus the fault schedule, without a browser (needs pip install cryptography, no CAIN code): for f in verify_pbft_qc_bundle.py verify_multi_region_bundle.py; do curl -so $f $f.txt; done; curl -so MULTI_REGION_BUNDLE.json PBFT_QC_BUNDLE.json; python3 verify_multi_region_bundle.py MULTI_REGION_BUNDLE.json