How to monitor an XDC node with xOne: metrics, logs, ethstats

How do I turn on the metrics endpoint?

Pass --metrics. The endpoint listens on 127.0.0.1 and port 6060 by default, and serves /metrics, /livez and /readyz.

xone --network apothem --datadir <datadir> \
     --metrics --metrics.addr 127.0.0.1 --metrics.port 6061 \
     --ethstats "<name>:<secret>@<host>:<port>"

The server answers three paths:

• /metrics returns Prometheus text.
• /livez returns 200 while the process answers. A partition does not make it fail, so it cannot cause a restart loop.
• /readyz returns 200 only when the node has a head, the head is not older than the limit, and the node is not syncing, degraded or shutting down. Otherwise it returns 503.

In the Rust node the head age limit is 60 seconds. The server accepts GET only.

Which metrics does an xOne node export?

The spec lists the families in the table below. They cover the head, sync state, peers, the txpool, queues, consensus rounds, storage and logging.

The Rust node has no head-hash metric. A label that changes every block makes one new series per block. To read the head hash, call eth_getBlockByNumber("latest", false).

What does an xOne node write to its logs?

An xOne node writes one line per event, at the level set by --verbosity, and limits the volume with budgets. The default level is 3 (info).

• Levels. --verbosity n sets the level: 0 silent, 1 error, 2 warn, 3 info, 4 debug, 5 detail. The default is 3.
• Per-module levels. --log.vmodule takes target=level pairs, such as xone_net or xone_sync::snap. The longest match wins.
• Format. The default line is <unix ms> LEVEL target message k=v. --log.format json writes one JSON object per line.
• Budgets. The global limit is 32 KiB/s with a burst of 128 KiB. ERROR lines use a reserved lane of 4 KiB/s with a burst of 16 KiB, so a flood of INFO lines cannot hide an error.
• Suppression. The node counts each dropped line. It writes at most one suppressed lines=N bytes=M line per subsystem every 10 s.
• Queue. The queue holds 8,192 lines. When it is full, the node drops the newest line and counts it. Logging never blocks a hot thread.
• File sink. A file sink rotates by size. The default is 64 MiB times 8 files.
• Slow blocks. --debug.logslowblock <duration> logs one JSON line with per-phase timings for each block that takes at least that long. It is off by default.

The spec page says the file path, size and count are not yet config keys. Under systemd, set the journald rate limit in the unit. The node cannot enforce it.

How does the xOne ethstats client work?

--ethstats name:secret@host[:port] makes the node report to a stats board. A URL without a scheme means wss://host/api. The node does not fall back to ws://. In xOneRS, the secret comes from ETHSTATS_SECRET in the env file, and there is no default. dumpconfig redacts it. A node in snap sync reports too (see the end of this section).

After login, the node reports every 15 s, and at once on a new head:

• node-ping and latency
• block: number, hash, parent hash, timestamp, miner, gas used, gas limit, transactions and roots
• pending: the pending transaction count
• stats: active, syncing, peers, gasPrice and uptime

The uptime value is the constant 100. After a failed login the client waits 1 s, then doubles the wait up to 60 s. Telemetry runs on its own thread, so a dead stats server does not slow block import.

PR 144 is merged. It starts the ethstats client when the node starts, so a node in snap sync shows on the board with its peers and the best header it has seen. This applies to xOneRS. This post does not cover the snap-sync behaviour of the other clients.

Does an xOne node send a SkyNet heartbeat?

Not yet. The spec describes a SkyNet heartbeat, and it is an open gap.

The spec describes a heartbeat that POSTs one JSON object every interval with three keys: blockHeight, peerCount and isSyncing. The keys use camelCase because SkyNet drops other spellings without an error.

The spec marks the payload types and the authentication as not verified, and the CLI reference lists no heartbeat flag.

Which signals should I alert on?

Alert on head age, peer count, free disk space, missed consensus turns, queue overflow and a down scrape target. The alert rules in xOneRS (ops/prometheus/alerts.yml) are a start.

• Head lag. Alert when xone_head_block_age_seconds is above 30 for 1 minute. This is the age of the head, not the distance to the network tip. To measure the distance, compare eth_getBlockByNumber("latest", false) with another node.
• Peers. Alert when xone_peers is below 3 for 5 minutes.
• Disk. Alert when the filesystem that holds the datadir has less than 15 % free for 10 minutes. The rule reads node_exporter's node_filesystem_avail_bytes and node_filesystem_size_bytes for the filesystem that prometheus.yml labels datadir="true". A second rule fires when that series is absent, so an unknown capacity is never read as healthy.
• Consensus. Alert on any increase of xone_missed_eligible_turns_total in 30 minutes, and when xone_timeouts_total grows by more than 10 in 15 minutes.
• Queues. Alert on any increase of xone_queue_overflow_total in 15 minutes.
• Scrape. Alert when the xone job is down for 2 minutes.

The repository also ships a Grafana dashboard, "xOne overview", and a local Prometheus and Grafana stack. Both ports bind to 127.0.0.1. For remote access, use an SSH tunnel or an authenticated TLS reverse proxy.

How do other clients describe monitoring?

Other clients document their own flags, ports and health checks. The table gives each claim as stated, with the xOne fact next to it. The claims are about other clients. They are not a like-for-like comparison.

Do the other xOne clients expose the same signals?

Mostly. The TypeScript, Python and C++ nodes serve /livez and /readyz and use the same default address and port, with fewer metrics. See which xOne client to run for the full comparison.

The xOneTS, xOnePy and xOneC repositories are private. All three use --metrics.addr 127.0.0.1 and --metrics.port 6060 as defaults, and serve /livez and /readyz.

All three still export xone_head_block_hash_info, which the Rust node removed. xOneTS and xOnePy use the same log budgets as the Rust node.

Frequently asked questions

What is the default metrics address of an xOne node?

The default address is 127.0.0.1 and the default port is 6060. Change them with --metrics.addr and --metrics.port. The endpoint serves GET only.

What is the difference between /livez and /readyz?

/livez returns 200 while the process answers. A partition does not make it fail, so it cannot cause a restart loop. /readyz returns 200 only when the node has a head, the head is not older than the limit, and the node is not syncing, degraded or shutting down. Otherwise it returns 503.

Does an xOne node report to ethstats during snap sync?

In xOneRS, yes. PR 144 is merged. It starts the ethstats client when the node starts, so a node in snap sync shows on the board with its peers and the best header it has seen. This post does not cover the other clients here.

How do I read the head hash of an xOne node?

Call eth_getBlockByNumber("latest", false). The Rust node has no head-hash metric, because a label that changes every block makes one new series per block.

How do I stop a log flood from hiding errors?

ERROR lines use a reserved lane of 4 KiB/s with a burst of 16 KiB, so a flood of INFO lines cannot hide an error. The global limit is 32 KiB/s with a burst of 128 KiB. The node counts each dropped line.