On 2026-11-15, net2040 produced no certified block from 08:59:06 to 12:34 UTC, about 3 hours 35 minutes. Block 852,529, the first block of epoch 952, never received a quorum certificate (QC), and no client in the network could build or vote around it. The chain resumed after a third re-proposal of a sibling block collected 11 of 14 votes. This entry follows the root-cause analysis in xOneRS#1345 and the step-by-step record on xOneRS#140. It is blameless. net2040 is a public devnet; nothing here describes mainnet or Apothem.

What happened?

REFERENCE TABLEScroll to explore all columns →
Time (UTC) Event
08:59:06 Block 852,529 (round 846,000, first block of epoch 952) sealed. It never gets a QC.
09:00:07 The Rust and TypeScript validators sign a timeout for round 846,000.
09:00:08 They receive block 852,529, 61 s after it was sealed.
10:16 Read-only diagnosis starts on #140.
11:08 setHead back to 852,528 on four geth-XDC nodes; it does not hold, because peers serve 852,529 again within 3 s.
11:24 Sibling 1 (round 846,145): 6 votes.
11:49 to 11:55 The Python, C++ and Rust seats move onto staged geth-XDC units, one key at a time.
11:57 TypeScript vote-path fix deployed.
12:08 Sibling 2 (round 846,189): 9 votes; the TypeScript seat never got the block.
12:13 One nethermind seat moves onto geth-XDC, with the owner's approval.
12:24 TypeScript sibling-delivery fix deployed.
12:34 Sibling 3 (round 846,215): 11 votes and a QC, carried by block 852,530.
12:38 Head at 852,572, all 14 seats sealing.

Why did block 852,529 get no QC?

A QC needs 10 of the 14 masternodes to sign the same digest. The votes were split or missing:

  • Gap number. A besu validator and two reth validators voted with the previous epoch's gap number (850,950 instead of 851,850), because their epoch-switch lookup fails for a block that is not yet on their canonical chain. Votes with a different gap digest do not combine into one QC.
  • Late block. The xOneRS and xOneTS validators received the block 61 s after it was sealed. They had already signed a timeout for that round, and refused to vote on a block of the same round ("safety unit refused a Timeout: round 846000 already signed with another digest"). The propagation gap is tracked as #1304.
  • No vote path. The xOnePy and xOneC validators did not vote: a block at the same height that does not extend their head never reached their vote logic.
  • Siblings. The nethermind validators do not vote for siblings; one never received the block.

Why could no client recover?

Recovery needs a leader that builds a sibling on the last certified block (852,528) and 10 validators that vote for a sibling of their own head.

REFERENCE TABLEScroll to explore all columns →
Client What stopped it
geth-XDC It builds only on the highest-QC block, but its miner remembers the parent of its last attempt and logs "already attempted block on this parent" every 2 s; only a restart clears it. debug_setHead did not hold.
xOneRS It unwound to 852,528 to build the sibling, a peer re-served 852,529 during the build, and the state view works only for the head, so the build failed with a misleading "outside the retained window" error (#1280). It also refused a second timeout for the same round with another gap digest, which geth sends (#1294).
xOneTS It dropped NewBlock and NewBlockHashes at or below its head height and never fetched a block its peers voted for.
xOnePy, xOneC They kept a V2 sibling as a side block that never reached the BFT machine.

A contributing rule: the Rust and TypeScript clients did not vote after their own timeout in the same round, while geth-XDC does (#1281).

How was the chain recovered?

  1. Re-propose. A geth-XDC validator restarted 240 s before the round it leads forgets its last parent and builds a sibling on 852,528. Three such re-proposals reached 6, 9 and 11 votes.
  2. Restore voters. The Python, C++ and Rust seats were moved onto staged geth-XDC units, one key at a time: stop the xOne signer, confirm no process holds the key, then start the geth unit. One nethermind seat moved the same way, with the owner's approval.
  3. Fix the TypeScript seat in place with two deployed fixes, so it accepted and voted for the sibling.

What was fixed?

Merged on 2026-11-15, as listed in the RCA:

  • xOneRS: lead on the highest-QC parent (#1289), vote after its own timeout as geth does (#1286), import a same-height sibling as a side block and vote by the voting rule (#1331).
  • xOneTS (private repository): the vote path (#96) and sibling delivery by NewBlock, hash announcement and fetch-on-vote (#98), both deployed to the TypeScript seat.
  • xOneC (private repository): the side-block hook posts the proposal (#100).
  • xOnePy (private repository): side blocks are offered to the BFT machine (#111).

All four xOne clients now vote for a sibling of their head.

What is still open?

From the RCA: the besu gap-number fix; sibling voting in nethermind; the xOneGo seat; sibling import in the Python follower and a db set-head command; executing before voting in the TypeScript and C++ clients; the 61 s propagation gap (#1304); the second-timeout refusal (#1294); moving the seats back from geth-XDC onto xOne; and a devnet liveness test that reproduces this failure.

Frequently asked questions

Why did net2040 stop on 2026-11-15?

Block 852,529 never got a QC: votes were split by a wrong gap number in some clients, and the xOne validators had signed a timeout before the late block reached them. Then no client could build or vote for a replacement block.

How long did the net2040 halt last?

About 3 hours 35 minutes, from 08:59:06 to 12:34 UTC.

Was any block lost or reorganized?

Block 852,529 and two earlier siblings were left without a QC and orphaned; the third sibling at the same height became canonical. No certified block was reverted.

Are the xOne seats back on xOne clients?

Not yet. Three seats run on staged geth-XDC units until the fixes are verified and the seats are moved back.

Sources