CAIN-42 MCPGate
Sign inGet started
Menu

CAIN-42 live cluster: network-partition test

Network-partition tests on the live CAIN-42 cluster cain-mr-01: 4 PBFT replicas on 3 hosts in 3 regions. For each test, one host's encrypted mesh link was taken down while its replicas kept running, then restored by a timer on that host. This page loads every quorum certificate each replica holds afterwards, and your browser re-checks them.

What happened

P1: a 1-replica host cut off (mia). The other three committed 6/6 writes. The isolated replica, still running, answered DENY, DENY to the writes sent to it and stayed at height 361 (it was 361): PASS.

P2: a 2|2 split. cain-mr-node-2, cain-mr-node-3 against cain-mr-node-1, cain-mr-node-4. Neither side has a quorum. Side A answered DENY, DENY and side B DENY, DENY. No replica committed anything during the split (no split brain): PASS.

After each heal all four replicas agreed again within about 3.4 s of the link returning. The identical decision chains below, on all four replicas, show there was no fork.

Method: ip link set cain42 down on the isolated host, healed by a timer on that host; isolated replicas were queried and sent writes from their own host while their mesh link was down.

Verify (about 5 seconds)

What is checked

  1. The membership configuration hash is recomputed from the 4 member ids and Ed25519 public keys. Every node and every certificate must carry it.
  2. For every sequence on every node, the COMMIT_QC and the PREPARE_QC. Each vote must be an Ed25519 signature by a distinct member over SHA-256 of the canonical signed message. It must have the right type (a COMMIT vote never counts as a PREPARE vote) and match this cluster, epoch, view, sequence and digest. Each certificate needs at least 3 distinct signers. The leader's proposal must be signed by the primary of that view, and its digest must bind the proposed operation.
  3. The certificate hash and signature-bundle hash are recomputed from the content.
  4. Evolution 3 fast path: a FAST_COMMIT_QC (a decision taken without the COMMIT round) is accepted only if the published membership declares the fast path and all four members signed it. Three of four is never enough for a fast commit.
  5. The decision chain is folded from genesis: decision_hash(seq) = H(cluster, epoch, seq, digest, parent). It must be contiguous and identical on all four nodes, and the nodes must end with the same application-state hash.
  6. View-change quorum certificates, and one consensus-to-enforcement AuthorizationCertificate per node.
  7. Negative controls: a certificate is tampered with in seven ways, and every tampered copy must be rejected.

The same checks, without a browser: curl -so verify_pbft_qc_bundle.py verify_pbft_qc_bundle.py.txt && python3 verify_pbft_qc_bundle.py PBFT_QC_BUNDLE.json (needs pip install cryptography, no CAIN code).