Why a one-block reorg wedged an xOne node on Apothem

What happened?

An Apothem validator running the Rust node (xOneRS) imported a competing block 85377493 and never left it. The issue lists the node as a validator, not a follower. Every later peer branch was refused, and the validator was penalized out of the set.

What was the impact?

The issue records this impact:

• Network: Apothem.
• Stuck block: the node imported block 85377493 (0x744488ae…). The canonical block was 0xf3b792dd….
• Refusals: the log line below appeared 681 times.
• Distance: the head stayed at 85377493, about 2,000 blocks behind.
• Consequence: the validator was penalized out of the set.

chain: setting peer … aside for 60 s: branch not adopted: accounts above 2^256 are held at the tip only

A restart did not help, and no rewind command existed.

What is the timeline?

The issue also records an interim recovery. The team kept the stuck datadir aside. The validator then synced again from an empty datadir with its safety record and node key carried over.

What was the root cause?

The node held balances above 2^256 only for the tip, with no way to restore them for an older block. So HistInner::can_pop_to refused every branch switch while any such account existed.

Apothem holds accounts with balances above 2^256. The wide-balance spec explains the data model. The EVM sees the balance modulo 2^256. The account leaf carries the true integer. A side map in a journal wrapper (WideJournal) tracks these accounts.

A branch switch pops blocks off the tip. To pop a block, the node must put back what that block changed. The issue says this worked for every other account. For wide balances there was no earlier value to put back. The code was already marked with a ponytail: comment that said a per-block undo is needed when a chain with such accounts needs a branch switch.

The issue notes that XDPoS V2 produces an ordinary one-block reorg on most timeout rounds. A refusal that is acceptable once becomes a permanent stop, because the node sets the peer aside, then meets the same refusal on the next branch.

85377492 ─┬─ 0x7444 (ours, orphan)  ← stuck: pop refused (wide balances)
          └─ 0xf3b7 (canonical) ─ … ─ tip

How did PR #606 fix it?

PR #606 records, for each block, the value that each changed wide account held before the block. pop_to then restores those values. Changes in hist.rs:

• HistInner::push stores a wide_undo entry for each block that changes a wide account. The entry holds the value before the block, or none if the account was not wide before.
• undo_from marks the first block this history pushed. Blocks below it came from the store and have no undo.
• pop_to restores wide balances from the undo, highest block first.
• trim drops the undo of blocks that leave the window.
• can_pop_to now refuses only when a popped block lies below undo_from while wide accounts are held. The error says the block has no undo.
• After a restart, the blocks replayed above the checkpoint have their undo again.

The PR also changes core.rs, which only initializes the two new fields.

Which tests guard the fix?

One unit test in crates/xone-node/src/services/hist.rs guards the fix: a_reorg_across_wide_balances_restores_them.

The test covers four cases:

1. It creates, changes and clears wide accounts over three blocks.
2. It runs a one-block reorg, checks that the balances return, and takes the other branch.
3. It pops deeper, back to before any wide account.
4. It checks that the pop is refused, with a "no undo" message, only below the first undo.

The wide-balance matrix test (crates/xone-evm/tests/wide_matrix.rs) guards the EVM side of wide balances. It does not test branch switches.

What remains open?

• No operator rewind. The issue asked for xone db rewind --to <n>. PR #606 lists it as not included. The xone db subcommands on main are stats and usage.
• Live evidence. The issue lists a net2040 reorg rehearsal, a live Apothem reorg, and a 24 hour fair-share soak with no refused branches. The issue and PR record no result for them.
• Limit below undo_from. A branch switch deeper than the replayed blocks is still refused while wide accounts are held.

For the sync context, see xOne sync modes. For run steps, see node operations and networks. For gate status, see the roadmap gates.

Frequently asked questions

Was the stuck node a validator or a follower?

Issue #577 calls it an Apothem validator. It was penalized out of the set after it stayed on the competing block.

Does a restart clear a wedge like this?

Not before PR #606. The issue states that a restart did not help and that no rewind command existed. The issue lists the interim recovery as a fresh sync from an empty datadir.

What does the fix not cover?

It does not add xone db rewind. It also refuses a switch below undo_from while wide accounts are held.

Why did PR #760 not merge?

PR #760 was an alternative fix for the same issue. It was closed with the note "Superseded by #606, already merged."