An XDC validator operator who runs xOne should check seven things every day: the Grafana dashboard, the alert state, the producer log lines, the seat, the stats board, the disk and the version stamp. Each item below has a short answer. The checklist is ours. Every item comes from a file in the xOne repositories.

This post only lists what to look at. For the setup, read how to monitor an XDC node with xOne. For the alert rules, read how to tune the xOne alerts. For the log formats, read xOne log formats. For the key and the hand-off steps, read key custody and seat rotation. The xOneTS repository is private.

Which Grafana panels do I check?

Open the dashboard "xOne overview" (uid xone-overview) and pick your instance. The repository ships one dashboard file, ops/grafana/xone-overview.json, with 16 panels:

  • Head block and Head age: the head rises, and its age stays small.
  • Peers and Required-block peer refusals: peers are present, and refusals do not climb.
  • Import rate (blocks/s) and Syncing: the node is not catching up.
  • Consensus round and highest QC round: both move together.
  • Round timeouts and Missed eligible turns: the second panel stays at 0. See the seat section.
  • Queue depth and Queue overflows: no overflow.
  • Storage and Watchdog lag: no jump.
  • Log lines dropped: no drops.
  • Txpool and RPC in flight by class: a glance only.

A test fails when the dashboard names a metric that the node does not export, so a renamed metric cannot leave an empty panel (monitoring.md).

Which alerts must be quiet?

All eight rules in ops/prometheus/alerts.yml must be quiet. Three are critical.

REFERENCE TABLEScroll to explore all columns →
Rule Fires when Severity
XoneHeadStale head older than 30 s for 1 minute critical
XoneMissedEligibleTurns any increase of missed eligible turns in 30 minutes critical
XoneMetricsDown /metrics does not answer for 2 minutes critical
XoneLowPeers fewer than 3 peers for 5 minutes warning
XoneQueueOverflow any queue overflow in 15 minutes warning
XoneRoundTimeouts more than 10 round timeouts in 15 minutes warning
XoneDiskAlmostFull datadir filesystem under 15 % free for 10 minutes warning
XoneDiskCapacityUnknown no datadir filesystem series for 10 minutes warning

XoneMissedEligibleTurns has no for: time, so it fires at once. If no alert fires, check that Prometheus still scrapes the node.

Which producer log lines do I look for?

Look for fence open once after each start, and sealed and sent for each block you seal. Producer lines start with producer: and go to stderr, also in JSON mode (producer.rs).

  • producer: fence open: the quiet window passed, signing starts. Before this line the node signs nothing.
  • producer: block <n> <hash> sealed and sent (seal <ms> ms, import <ms> ms). A large seal time shows a stalled disk.
  • No fence closed: our key signed .... This line means your key signed in another process, and the quiet window restarts.
  • No stopping: ... line and no slow safety write: a BFT step took <n> ms.
  • Few round <r> not proposed: ... and block <n> (round <r>) not produced: ... lines.

A timeout for round <r> sent line is normal when another validator misses its turn. Count those lines against Round timeouts.

How do I watch the seat?

Compare your eligible turns with your sealed blocks, and give every miss a cause. The mint gate computes the turns from round % 900 % N on the masternode list in force (xOneTS mint-gate.md).

  • Misses with a cause. The mint gate records each miss as one of: candidate fault, network or peer, other validator, host or environment, unknown. A miss with cause unknown blocks the gate.
  • Seal checks. A seal counts when the block is stamped after the flip, the next block names it as parent and is canonical on an independent node, and your own log has the seal line with the same number and hash.
  • Abort rule. On a missed turn, a wrong-root error, a rejected own block, or a vote on two blocks of one round: stop the producer at once, return the seat and record what happened. Stop within 45 to 60 seconds. Do not wait out a 300 s dead slot.
  • Idle seat. A masternode that seals no block in a whole epoch is penalised at the next epoch switch and leaves the list. One epoch is 900 rounds on net2040. Read penalties and comeback for the rules.

Zero misses is an observation, not a guarantee. With n eligible turns and 0 misses, the 95 % upper bound on the true miss rate is about 3/n.

Does the stats board show my node?

It should, if you pass --ethstats. The node reports every 15 s and at once on a new head. The stats frame carries active, syncing, peers, gasPrice and uptime (ethstats.md).

  • The node is listed, and its block number matches the other nodes.
  • The role reads validator. The login info says validator only when the node is authorised and sealing. Otherwise it says follower and carries the zero address.
  • Do not read uptime. It is the constant 100.

A node in snap sync also reports, since PR 144.

Which disk lines matter?

Watch for disk_low, disk_ok and disk_full, and check the free space of the datadir filesystem. The node writes them to stderr (diskguard.rs).

  • xone-node: disk_low free=<n> stop_below=<n>; block import stopped. Import and sealing stop at a block boundary.
  • xone-node: disk_ok free=<n>; block import resumed.
  • xone-node: disk_full free=<n> exit_below=<n>; stopping the node. The exit code is 4 (XONE-E005, errors.md).

The defaults stop import at 2 % or 2 GiB free, resume at 3 % or 3 GiB, and exit at 1 % or 1 GiB (config.rs). The XoneDiskAlmostFull alert at 15 % is the early warning. xone db usage shows where the space goes. See XDC node hardware and disk requirements.

Does my version stamp match the release?

Run ops/xone-version.sh <instance>. It compares the installed binary's xone --version, the running process and web3_clientVersion. It exits 0 when all match, 1 on a mismatch or stale process, and 3 when a stamp could not be read. It changes nothing (xone-version.sh).

  • The stamp has the form xOne/v<V>-<commit8>-<DDMMYYYY>/linux-x86_64/rustc1.94.0, with no unknown and no -dirty (install.md).
  • The metric xone_build_info{version} carries the same string as web3_clientVersion and the ethstats node field.
  • The version equals the tag you meant to run. The latest release is v0.1.0, published 2026-11-12.

A producer runs a tag only after a follower soaked it for at least 24 h across at least one epoch switch (release.md).

How do other clients describe this?

Other clients document dashboards and health checks in their own terms. The rows are claims as stated, not a like-for-like test.

REFERENCE TABLEScroll to explore all columns →
Client Claim, as stated xOne on main
reth Shipped Grafana dashboard. Source: reth monitoring docs, no date listed Dashboard "xOne overview" with 16 panels, and alerts.yml with 8 rules
nethermind /health returns 503 on NoPeers. A low-disk shutdown threshold. Source: Nethermind health check docs, no date listed /readyz returns 503 when the node has no head, a stale head, or is syncing. The disk guard exits at 1 % or 1 GiB free by default.

Frequently asked questions

What is the first thing to check each day?

Check that no alert fires and that the Missed eligible turns panel is at 0. XoneMissedEligibleTurns is critical and fires at once.

Which log line shows that my validator sealed a block?

producer: block <n> <hash> sealed and sent (seal <ms> ms, import <ms> ms). Check the same number and hash on an independent node.

What does fence closed mean?

Your key signed in another process. The node restarts its quiet window. Check at once that exactly one process holds the key.

When must I stop the producer?

Stop it on a missed turn, a wrong-root error, a rejected own block, or a vote on two blocks of one round. Stop within 45 to 60 seconds.

How do I know I run the release I installed?

Run ops/xone-version.sh <instance>. Exit code 0 means the installed binary, the process and web3_clientVersion match.

Sources