How to tune Prometheus alerts for an xOne node on XDC

Where are the rules, and how do I load them?

The rules are in ops/prometheus/alerts.yml. Load the file as a rule file in Prometheus and route it through your Alertmanager. The local stack in ops/grafana/docker-compose.yml already does the first step (monitoring.md).

prometheus.yml sets a 15 s scrape interval. After an edit, run cd ops/prometheus && promtool test rules alerts_test.yml. It tests the two disk rules.

What does the datadir="true" label do?

It marks the one filesystem that holds the datadir, so the disk rules need no mount path and no join between targets. The node scrape job in prometheus.yml sets the label with one regex on mountpoint. node_exporter sees the host root under /host.

The default regex matches /host, /host/var/lib/xone and /host/data. Set it to the mount of your datadir:

     metric_relabel_configs:
       - source_labels: [mountpoint]
-        regex: "/host(/var/lib/xone|/data)?"
+        regex: "/host<datadir-mount>"
         target_label: datadir
         replacement: "true"

What does XoneHeadStale tell me?

expr: xone_head_block_age_seconds > 30
for: 1m
severity: critical

The head block is more than 30 s old for 1 minute. XDC makes a block every 2 s. The metric does not exist while the node has no head, so the rule stays silent then. The expression has no xone_syncing term. A node that catches up with an old head can match it.

Check first: curl <metrics-addr>/readyz. A 503 body names every cause, such as not ready: stale_head syncing. The causes are no_head, stale_head, syncing, degraded and shutting_down. Knob: the 30 and the for:. The /readyz age limit is a separate setting.

What does XoneLowPeers tell me?

expr: xone_peers < 3
for: 5m
severity: warning

The node has fewer than 3 peers for 5 minutes. Check first: the bootnodes. The default is bootnodes/<net>.txt, and BOOTNODES in the env file replaces it. The counters xone_p2p_disconnects_total{reason,side} and xone_peers_required_block_refused_total show why sessions end (metrics.rs). Knob: the 3 and the 5m.

What does XoneMissedEligibleTurns tell me?

expr: increase(xone_missed_eligible_turns_total[30m]) > 0
severity: critical   # no for:

An eligible mining turn of this node produced no block in the last 30 minutes. The rule has no for:, so it fires at once. In the Gate D rehearsal, on a private devnet, restarts and a 60 s partition caused misses. In one run, 30 of 31 misses fell inside restart windows. After kill -9 the node replayed from its last state checkpoint for about 60 s. Check first: whether the node restarted, and /readyz. Knob: the window 30m and the > 0. Silence the rule in Alertmanager during a planned restart.

What does XoneQueueOverflow tell me?

expr: increase(xone_queue_overflow_total[15m]) > 0
severity: warning   # no for:

A bounded queue hit its limit and took its overflow action in the last 15 minutes. The queue label is Q1 to Q18. ADR-007 lists the queues, their bounds and the overflow action of each. Check first: the queue in the alert summary, then its row in ADR-007. Some queues drop on purpose. Q1 holds votes and drops the oldest. To ignore one queue, add a queue matcher to the selector.

What does XoneRoundTimeouts tell me?

expr: increase(xone_timeouts_total[15m]) > 10
severity: warning   # no for:

More than 10 consensus rounds timed out in 15 minutes. A round with no canonical block is a timeout of its leader. In the Gate D rehearsal a geth validator stopped for 60 s, and its rounds timed out. In a run of 3,656 blocks, all 31 timeouts were rounds this node missed. Check first: xone_consensus_round against xone_highest_qc_round, and the peer count. Knob: the 10 and the 15m.

What does XoneDiskAlmostFull tell me?

expr: >
  node_filesystem_avail_bytes{datadir="true"}
    / node_filesystem_size_bytes{datadir="true"} < 0.15
for: 10m
severity: warning

The datadir filesystem has less than 15 % free for 10 minutes. Space used by other software on the disk counts. Check first: xone db usage shows where the space goes. Knob: the 0.15 and the for:. This alert is the early warning. The node has its own guard in the [limits] config table: it stops import at 2% or 2 GiB free and exits with code 4 at 1% or 1 GiB. Set limits.disk_stop_pct and the other disk_* keys to move it.

What does XoneDiskCapacityUnknown tell me?

expr: absent(node_filesystem_avail_bytes{datadir="true"})
for: 10m
severity: warning

No series carries datadir="true". Either node_exporter is down or the regex matches nothing. Check first: up{job="node"}, then the mountpoint regex. Knob: the regex, as shown above.

What does XoneMetricsDown tell me?

expr: up{job="xone"} == 0
for: 2m
severity: critical

Prometheus cannot scrape /metrics for 2 minutes. Check first: the node runs with --metrics, and the target matches --metrics.addr and --metrics.port (default 127.0.0.1:6060). The local stack needs --metrics.addr 0.0.0.0 and the target host.docker.internal:6060. Another process on the port gives start-up error XONE-E007. Knob: the target in prometheus.yml, or ops/prometheus/xone-targets.sh set <name> <host:port> [network] for more nodes. remove <name> stops the scrape.

How do other clients describe alerting?

The claims are as stated by each project.

Frequently asked questions

Which xOne alerts are critical?

XoneHeadStale, XoneMissedEligibleTurns and XoneMetricsDown are critical. The other five rules are warning.

Why does XoneDiskCapacityUnknown exist?

When node_exporter is down or the regex matches nothing, no free-space series exists. The rule reports that case, so unknown capacity is never read as healthy.

How do I change the disk alert threshold?

Change 0.15 in XoneDiskAlmostFull, then run promtool test rules alerts_test.yml from ops/prometheus. To change when the node stops import, set the limits.disk_* keys in the config file.