On 2026-11-15 at 04:05:22 UTC, the xOnePy validator on net2040 missed one turn: round 841,894. Block 848,424 went to the next leader. The node did not stop or restart, and the next turn sealed. The recorded cause is host load: the load average was 41.8 on 20 cores, and the node's single event loop woke late. This note gives the record, the mechanism and the lesson. It is blameless and uses only the soak record on xOnePy#85 (private repository) and the xOnePy source.
net2040 is a public devnet. Nothing here describes mainnet. For the week's other entries see the net2040 incident log, week 46.
What happened?
| Item | Recorded value |
|---|---|
| Time | 2026-11-15 04:05:22 UTC |
| Round | 841,894 |
| Block | 848,424, sealed by the next leader |
| Consecutive misses | 1 |
| Seat tally at that point | 867 of 869 eligible turns sealed since the flip of 2026-11-14 13:03 UTC |
| Host load average | 41.8 on 20 cores (compile jobs at nice 19, plus other services) |
| Node log | watchdog: heartbeat late by 195 ms and 245 ms (shedding load); event loop stall 204 ms |
| 30 s later | 15 peers, 3 blocks behind, process alive, no restart |
| Classification | Host or environment. Not a candidate fault, not a network fault |
Action: none on the node. The miss counts in the soak. The fast-abort rule applies to a borrowed seat; this was the node's own seat soak, and the next turn sealed.
Why does host load make a Python validator miss a turn?
xOnePy runs its node on one asyncio event loop. Two parts of the limits module watch that loop (limits.py, private repository):
- The lag probe sleeps 50 ms on the loop and measures how late it wakes. A wake more than 100 ms late is logged as
event loop stall N ms. Each wake is also the heartbeat. - The watchdog runs on its own thread and reads the heartbeat. Its levels: more than 100 ms late is a warning (
heartbeat late by N ms (shedding load)), more than 2 s is an error, more than 10 s is fail-stop.
When the host is busy, the loop thread gets less CPU time, and every task on the loop runs later, the block work included. At round 841,894 the loop woke 204 ms late and the heartbeat was 195 to 245 ms late. That is well inside the warning level, so the node kept running, but the turn passed to the next leader. The record does not split the time between building, signing and sending the block.
Has this happened before?
Yes, the same seat recorded three earlier host-load events on 2026-11-14:
- 06:49:47 UTC, round 824,976: a single miss at load average about 130.
- 12:35 to 13:02 UTC: an outage of about 19 turns at load 50 to 75 with the disk 99 to 100% full. The node stopped and had to be relaunched.
- 19:05:20 UTC, round 834,666: a single miss at load 55 from 21 compile processes.
This time the node stayed up, logged the late heartbeat at warning level, and sealed its next turn.
What is the lesson?
- Keep builds on a validator host at nice 19 and few jobs. The builds at round 841,894 were already at nice 19, and the loop was still late by 204 ms. A lower priority helps, but it does not remove the contention for a single event loop. Fewer parallel jobs, or a different host for builds, is the next step.
- Watch the stall signal, not only the miss counter. The stall and late-heartbeat lines came at the moment of the miss. Alert on
event loop stallandheartbeat latelog lines on a validator. - Do not rely on the lag gauge alone.
xone_loop_lag_msis the last 50 ms sample of the probe. A scrape every 15 s sees a 204 ms stall only if it is the last sample before the scrape. There is no stall counter yet; xOnePy#108 (private repository) asks forxone_loop_stalls_totaland a late-heartbeat counter.
Frequently asked questions
Did the Python validator crash at round 841,894?
No. The process stayed alive with 15 peers, did not restart, and sealed its next turn.
What was the cause of the missed turn?
Host load. The load average was 41.8 on 20 cores, the event loop woke 204 ms late, and the turn passed to the next leader.
Is 867 of 869 a good soak result?
It is the recorded tally since the 2026-11-14 13:03 UTC flip: two misses, both recorded with host load as the cause.
Which metric shows event-loop stalls in xOnePy?
Today only the log lines and the xone_loop_lag_ms gauge, which is the last sample. A counter is requested in xOnePy#108.
Sources
- xOnePy#85 soak records of 2026-11-14 and 2026-11-15 (private repository)
- packages/node/src/xonepy_node/limits.py and metrics.py (private repository)
- xOnePy#108 (private repository)
This article describes the source and evidence at its review date. Check the linked repository for current implementation and qualification status.
Explore the engineering evidenceEdit this page on GitHub opens the Markdown source (content/blog/python-validator-load-miss/index.md). GitHub may ask you to sign in. Report a correction opens a prefilled GitHub issue for this page.