Alert on ten series from the xOne /metrics endpoint: head age, syncing, peers, required-block refusals, missed turns, round timeouts, queue overflow, storage bytes, dropped log lines and slow blocks. This post gives the exact name, type and meaning of each one, one alert expression and the first thing to check.
It does not repeat the rule-by-rule tuning in how to tune xOne alerts or the endpoint set-up in how to monitor an XDC node with xOne. Names, types and help text come from metrics.rs in xOneRS on main.
Which of these series does the node fill today?
Five series carry live values on main: head age, syncing, peers, required-block refusals and slow blocks. The other five are exported, but the node does not set them yet, so they read 0.
The code that builds the snapshot says it fills "the parts of the metrics snapshot this node knows; the rest stays zero until its source is wired" (stats.rs). It names no field for timeouts, missed turns, storage bytes, queues or log counters. The queue families have no queue rows then. Issue #994 tracks the wiring; until it lands, three shipped rules (missed turns, timeouts, queue overflow) cannot fire. Before you rely on an alert, run curl <metrics-addr>/metrics and look at the series.
1. How old is the head block?
xone_head_block_age_seconds is a gauge: "Seconds since the head block timestamp." XDC makes a block every 2 s. The series is absent while the node has no head.
expr: xone_head_block_age_seconds > 30
for: 1m
This is the shipped rule XoneHeadStale (alerts.yml). When it fires, call /readyz. The 503 body names the cause.
2. Is the node syncing?
xone_syncing is a gauge: "1 while syncing." The code reads it as "behind the best peer". No shipped rule uses it. This one is an example, not a shipped rule:
expr: min_over_time(xone_syncing[30m]) == 1
It fires when the node reports that it is syncing (behind the best peer) for the whole 30 minutes. Pick your own window. A new node syncs on purpose. Check the peer count and the head age next.
3. How many peers does the node have?
xone_peers is a gauge: "Connected peers."
expr: xone_peers < 3
for: 5m
This is the shipped rule XoneLowPeers. Check the bootnodes first.
4. Did the node drop peers on the wrong chain?
xone_peers_required_block_refused_total is a counter: "Peers disconnected for another hash at a required block (--eth.requiredblocks)." The checks are in peer chain verification. Apothem has one built-in required block. The other networks have none unless you pass --eth.requiredblocks. No shipped rule uses it. Example:
expr: increase(xone_peers_required_block_refused_total[1h]) > 0
A refusal is normal while stale peers exist. A steady rise with a low xone_peers means your peer set is on another branch. Check the peer refused log lines.
5. Did an eligible mining turn produce no block?
xone_missed_eligible_turns_total is a counter: "Eligible mining turns that produced no block from this node."
expr: increase(xone_missed_eligible_turns_total[30m]) > 0
This is the shipped rule XoneMissedEligibleTurns, severity critical. Check for a restart first. It applies to a node that mines.
6. How many rounds timed out?
xone_timeouts_total is a counter: "Round timeouts."
expr: increase(xone_timeouts_total[15m]) > 10
This is the shipped rule XoneRoundTimeouts. See QC, TC and timeouts in XDPoS v2 for the mechanism. Check the peers and the other masternodes.
7. Did a bounded queue overflow?
xone_queue_overflow_total{queue} is a counter: "Overflow actions per ADR-007 queue." The queue label is Q1 to Q18.
expr: increase(xone_queue_overflow_total[15m]) > 0
This is the shipped rule XoneQueueOverflow. Read the label in the alert, then the queue's row in ADR-007.
8. How big is the datadir?
xone_storage_bytes is a gauge: "Bytes used by the datadir." The shipped disk alert does not use it. It reads free space from node_exporter. Use this series for growth. Example, with your own limit:
expr: xone_storage_bytes > <limit-bytes>
Check xone db usage for what takes the space. For sizing, read disk planning for xOne.
9. Is the log losing lines?
xone_log_dropped_lines_total is a counter: "Log lines dropped by a full sink queue." The queue holds 8,192 lines. When it is full, the node rejects the newest line (logs.rs). The shipped Grafana panel plots increase(xone_log_dropped_lines_total[15m]). As an alert, use the same expression:
expr: increase(xone_log_dropped_lines_total[15m]) > 0
Check the log sink and the disk it writes to. The sister series xone_log_suppressed_lines_total counts lines the rate limits drop.
10. Are blocks slow to import?
xone_slow_blocks_total is a counter: "Imported blocks at or above the --debug.logslowblock threshold." The flag is off by default (-1ns), so the series stays at 0 until you set a threshold (flags.md). Example:
expr: increase(xone_slow_blocks_total[15m]) > 0
Each slow block also writes one JSON log line with the time of the verify, senders, execution, commit, history and total phases. Read that line to see which phase is slow.
How do other clients expose metrics?
The claims are as stated by each project.
| Client | Claim, as stated | xOne on main |
|---|---|---|
| geth | --metrics, --metrics.addr, --metrics.port (6060), Prometheus path /debug/metrics/prometheus (docs, 2024-12-16) |
same three flags, port 6060, path /metrics |
| reth | --metrics 127.0.0.1:9001, a shipped Grafana dashboard (docs) |
a shipped Grafana dashboard and alert file |
| besu | --metrics-enabled, port 9545, Grafana dashboard 16455 (docs) |
--metrics, port 6060 |
| nethermind | /health endpoint, 503 on NoPeers (docs) |
/readyz returns 503 with named causes |
Frequently asked questions
Which xOne metric shows that the node is stuck?
xone_head_block_age_seconds. The shipped rule fires when it stays above 30 for 1 minute. XDC makes a block every 2 s.
Does an alert on xone_timeouts_total work today?
The metric exists, but the node does not set it yet, so it reads 0. Check the series with curl before you add the alert.
Is there a head hash metric?
No. A label that changes every block makes one new series per block. Call eth_getBlockByNumber("latest", false) instead.
Why is xone_slow_blocks_total always 0?
--debug.logslowblock is off by default. Set a duration, and the counter counts each imported block at or above it.
Sources
This article describes the source and evidence at its review date. Check the linked repository for current implementation and qualification status.
Explore the engineering evidenceEdit this page on GitHub opens the Markdown source (content/blog/xone-metrics-ten-series/index.md). GitHub may ask you to sign in. Report a correction opens a prefilled GitHub issue for this page.