When four XDC clients share one host, CPU load and a full disk can stop a validator before any consensus rule does. On 2026-11-14 the xOnePy seat on net2040 stopped for about 27 minutes for that reason. The records below show the load, the disk state and the limits that were in play.
This post lists records, not advice. For the disk lines and sizes, read disk planning and hardware and disk. xOneTS, xOnePy and xOneC are private repositories.
What happened to the Python seat on 2026-11-14?
The Python (xOnePy) seat stopped sealing from 12:35:26Z to 13:02:55Z. The cause was the host, not consensus, as recorded in xOnePy#85 (comment of 13:19Z).
- The host load average was 50 to 75. The data disk was at 99 to 100% from build directories.
- The node's watchdog saw its heartbeat late by 0.9 to 1.9 s for 20 ticks. It stopped the node with exit 4 at 12:35:26Z.
- The loop restarted the node as a follower. The automatic rollback to the geth signer failed, because another process held the signer's port as the source port of an outbound connection. The seat had no signer.
- At 12:49:59Z the disk exit line stopped the node, and it stopped it again on the relaunch.
- After 240 GB was freed, the node restarted in standby with the key. Mining started at 13:02:55Z (flip block 836393). About 19 eligible turns were missed.
Over the whole 14-seat era, the Python seat sealed 1,807 blocks (7.12%) against a fair share of 1,813. The two geth seats at 7.24% picked up the lost turns (xOneRS#140, comment of 17:55Z). See the fair-mint measurement.
How high was the host load during the runs?
The recorded load averages were 14 to 76 on a shared 20-core build host. Every source below states the load beside the result.
| Client | Record | Load average |
|---|---|---|
| xOneRS | Apothem genesis to 766,981 in 108 minutes, workspace tests running beside it (apothem-follow.md) | 17 to 27 |
| xOneRS | Build times, called "upper bounds" by the page (ADR-002) | 38 to 76 |
| xOneRS | Post-quantum cost lab: 41 of 51 rows over tolerance (cost-lab-plan.md) | 14 to 72 |
| xOneTS | Longest event-loop gap in a block benchmark: 180 to 550 ms (net2040-ts-node.md) | 25 to 42 |
| xOnePy | Full sync from peers (py-performance.md) | 34 to 50 |
| xOneC | "The host is never quiet" (performance.md) | 31 to 48 |
The xOnePy page also sets a rule. A committed benchmark judges its budget only at a load of at most 0.5 per core. Above that, it prints the numbers and skips the assertion.
The producer seat saw higher figures: a load average of about 130 at the 06:49Z miss, and 55 with 21 concurrent rustc and cargo processes at the 19:05Z miss (both in xOnePy#85). The record names the cause of both: build lanes on the same host.
Does the load average predict the damage?
Not alone. The xOneC record of a safety-record write shows it (ADR-010 addendum). One run had a RocksDB build at -j8: fsync p99 was 129.820 ms at load 30. A second run had no build of ours: p99 was 57.515 ms at load 33 to 38.
The ADR says the tail comes from other load on the shared disk, not from the record. Disk contention shows in p99 before it shows in the load average.
What did the disk lines do to the node?
On a full shared disk, the node's own exit line ended the run. The clients do not use the same lines.
| Client | Disk-pressure action | Source |
|---|---|---|
| xOneRS | Stop import below 2% or 2 GiB free. Exit code 4 below 1% or 1 GiB, two samples in a row. | ADR-007, config.rs |
| xOnePy | Stop below the larger of 10% and 20 GiB. Exit after two samples below the larger of 2% and 4 GiB. --datadir.minfreedisk replaces the three lines. |
limits.py |
| Nethermind | A low-disk shutdown threshold, LowStorageSpaceShutdownThreshold, as stated in the health-check docs. |
no date listed |
The xOneTS addendum adopts the 10% and 20 GiB proposal (ADR-007 addendum). The xOneC addendum stops at 85% used and says a node on its 99%-used build host "would be stopped at start" (addendum).
The xOneRS defaults come from one build host with 41 GB free of 420 GB. The proposed 10% line stopped every follower test there, so the code uses 2% and 2 GiB. Hardware and disk tells this story.
The xOnePy record lists two follow-ups: a validator should shed load and keep sealing on a late heartbeat, and a disk warning should not become a seat outage. The record says pausing import while the signer stays alive is the geth behaviour. This post does not claim a fix for either.
How much memory does each client hold?
At one probe, the xOne processes held 83,408 to 298,696 kB (VmRSS) on net2040 and 179,856 to 10,107,556 kB on Apothem. This is one sample, with different modes and networks. Do not read it as a ranking.
Source: xOne team measurement, 2026-11-14 19:20 UTC. xOnePy runs extra worker processes (the sender-recovery pool has 18) that the probe does not count.
| Network | Client and mode | VmRSS kB | RssAnon kB | RssFile kB |
|---|---|---|---|---|
| net2040 | xOneRS full | 85,948 | 65,116 | 16,492 |
| net2040 | xOneRS archive | 198,324 | 88,216 | 108,164 |
| net2040 | xOneTS full | 298,696 | 277,000 | 24,864 |
| net2040 | xOneC full | 106,064 | 58,476 | 46,888 |
| net2040 | xOnePy snap | 83,408 | not read | not read |
| Apothem | xOneRS snap | 519,008 | 489,112 | 33,192 |
| Apothem | xOnePy snap | 179,856 | 166,264 | 14,236 |
| Apothem | xOneC snap | 10,107,556 | 349,504 | 9,793,144 |
RssAnon is heap. RssFile is mapped files, such as an MDBX map. The xOneC Apothem row is almost all mapped file. The two values were read a few minutes after VmRSS, so their sum can differ from it.
The xOneRS systemd unit sets MemoryMax=15G. Its comment calls it a cap from the xOneGo operator record, a 30 GB blow-up on a shared host, and says "Tune per host" ([email protected]).
What did the team change on the shared host?
After the 19:05Z miss, the record says the workspace gate was stopped and the build lanes run at nice 19 with -j 2. The node kept mining. One sample at one time is not a trend. For the alerts and the node's own checks, see node operations.
Frequently asked questions
Why did the Python seat stop on 2026-11-14?
The host was at load 50 to 75 with the data disk at 99 to 100%. The watchdog stopped the node at 12:35:26Z, and the rollback to the geth signer failed. Mining restarted at 13:02:55Z after 240 GB was freed.
Did the outage break the Python client's fair share?
No. Over the 14-seat era the Python seat sealed 1,807 blocks (7.12%) against a fair share of 1,813. All xOne seats were within 0.5% of their share.
Do all four xOne clients use the same disk lines?
No. xOneRS uses 2% or 2 GiB to stop and 1% or 1 GiB to exit. The xOnePy code uses 10% and 20 GiB to stop and 2% and 4 GiB to exit. xOneC's addendum stops at 85% used.
What does MemoryMax do in the xOneRS unit?
It caps the service at 15G. The unit says the value comes from a 30 GB blow-up on a shared host and must be tuned per host.
Sources
- xOnePy: issue #85 (comments of 2026-11-14T06:49Z, 13:19Z and 19:05Z), limits.py, py-performance.md, ADR-007 addendum. Private repository.
- xOneRS: issue #140 (comments of 2026-11-14T16:55Z and 17:55Z), ADR-007, ADR-002, apothem-follow.md, cost-lab-plan.md, [email protected].
- xOneTS: net2040-ts-node.md, ADR-007 addendum. Private repository.
- xOneC: performance.md, ADR-010 addendum, ADR-007 addendum. Private repository.
- Nethermind: health-check docs.
- Memory table: xOne team measurement, 2026-11-14 19:20 UTC, one sample.
This article describes the source and evidence at its review date. Check the linked repository for current implementation and qualification status.
Explore the engineering evidenceEdit this page on GitHub opens the Markdown source (content/blog/four-clients-one-host/index.md). GitHub may ask you to sign in. Report a correction opens a prefilled GitHub issue for this page.