An xOne node writes few lines that need a decision. Ten of them cover start-up, sync, a bad peer, a full disk, block sealing and shutdown. This post gives the exact format of each line, from the xOneRS source on main, what it means and what to do.

The shape of text and JSON logs is in xOne log formats. Metrics and alert rules are in how to monitor an xOne node. This post does not repeat either.

Which stream carries these lines?

Most lines below use eprintln! in the source, so they go to stderr. The started and stopped lines go to stdout. Collect both streams.

In text mode the lines look as shown. The --log.format flag does not change the stderr lines. In JSON mode, started and stopped become log objects (main.rs:339-346).

1. What does the start-up line say?

The node prints one line after every service has started. It names the chain, the genesis hash, the datadir, the sync mode and the ports that are bound (main.rs:422).

started chain <chain id> genesis <hash> datadir <datadir> sync <mode>; serving <kind> <addr>, ...

Do: check that the chain id and genesis hash are the network you meant. If serving says no listeners, no port is open (endpoints.rs:25-33). The build stamp is not in this line. xone --version prints it (main.rs:61).

2. How do I know a peer connected?

This line shows that a peer finished the handshake and the follower now knows it (chain.rs:783).

chain: peer <short id> connected, eth/<version>, rank <rank>

Do: nothing, if you see it. If the node stays at head with no such line, check the xone_peers gauge and your bootnodes.

3. How do I see sync progress?

The follower prints one line for each batch it imports (chain.rs:607). A batch of one block prints too, so at the tip you see one line per block.

chain: imported <n> blocks (<txs> transactions) in <ms> ms, head <number>

Do: read head as the new head block number. No line for a long time while peers are connected means no import. Compare xone_head_block_age_seconds (monitoring). A reorg prints a different line: chain: adopted the branch of peer <id>: <n> block(s) of ours dropped, <n> imported, head <n> (chain.rs:551).

4. What does it mean when a peer is set aside or refused?

The follower sets a peer aside for a fixed time when a request fails (chain.rs:411). A block that fails to import is reported the same way: the error text is the reason.

chain: setting peer <short id> aside for <s> s: <why>

The usual time is 60 s (chain.rs:59). A peer that is only behind is set aside for 4 s (does not know our head yet) (chain.rs:56-63). Other reasons in the source are request timed out, headers do not link, no bodies and answered from another block.

If you set --eth.requiredblocks, the network layer drops a peer that serves another hash at a listed height, with its own line (peers/mod.rs:856):

peer refused  id=<short id>  reason=required-block  number=<n>  want=<hash>  got=<...>

Do: one line is normal. Many lines with the same reason across peers point at your node or your network. A required-block refusal means the peer is on another chain or fork. Do not remove the flag to quiet it.

5. What do disk_low, disk_ok and disk_full mean?

The disk guard samples free space on the datadir filesystem. It closes the import gate below the stop line, opens it above the resume line and stops the node below the exit line (diskguard.rs:152-163).

xone-node: disk_low free=<bytes> stop_below=<bytes>; block import stopped
xone-node: disk_ok free=<bytes>; block import resumed
xone-node: disk_full free=<bytes> exit_below=<bytes>; stopping the node

The default lines are 2 % or 2 GiB to stop, 3 % or 3 GiB to resume and 1 % or 1 GiB to exit. The larger of the two numbers wins (config.rs:401-407, 570-577). The node samples every 5 s by default. After disk_full the exit code is 4 (lifecycle.rs:40-47).

Do: free space or grow the disk. Import resumes by itself after disk_ok. A producer also stops sealing while the gate is closed (producer.rs:1130). Plan size first: disk planning for xOne.

6. When does the producer print "fence open"?

A node that mines does not sign at start. The fence counts seconds of connected time with valid gossip. When it opens, the producer prints one line (producer.rs:433).

producer: fence open: the quiet window passed, signing starts

Do: if you expect sealing and the line never comes, check peers and gossip first. The rule is in ADR-010.

7. What does "sealed and sent" tell me?

The node sealed a block and sent it to its peers (producer.rs:1196).

producer: block <number> <hash> sealed and sent (seal <ms> ms, import <ms> ms)

The seal time covers the safety-record write, its sync and the signature. A stalled disk shows as a large seal number (producer.rs:1124-1127). A refusal from the safety unit has its own line:

producer: the safety unit refused a <kind>: <why>

Do: watch the seal value for a trend. On a refusal, the node did not sign. Read <why> (producer.rs:1338).

8. What is a "suppressed" line?

The log budget dropped lines and the node says how many (logs.rs:105).

suppressed lines=<n> bytes=<n> subsystem=<name>

Do: treat the log of that subsystem as incomplete for that window. Alert on xone_log_suppressed_lines_total (metrics.rs:281). The budgets are in log formats.

9. What is the "Slow block" line?

It is one JSON object with "msg": "Slow block" and the phase times of one import (slow.rs:57). It is off unless you set --debug.logslowblock.

Do: read timing to see which phase is large: verify, senders, execution, commit or history. Count the lines with xone_slow_blocks_total. The full shape is in log formats.

10. What do the stop lines say?

On a clean stop the follower writes a state checkpoint, then the node prints stopped (chain.rs:1025, main.rs:433).

chain: state checkpoint of block <n> written
stopped

The failure forms are chain: no checkpoint written: <e> (chain.rs:1027) and store: the last blocks are not steady on close: <e> (core.rs:475). If a critical worker ends, the node prints node: critical worker <name> ended: <reason>; stopping the node and exits with code 5 (lifecycle.rs:109).

Do: after stopped, restart as usual. After a kill, a restart replays what is stored (main.rs:288-289).

What is "unsupported history", and is it a log line?

It is not a log line. It is a JSON-RPC error (code -32000) for data outside the retained window (types.rs:82). The message names the first retained block and the pivot (retention.rs:50-52). The log line that is related is rpc: the history of block <hash>: <e>, which reports a failed history read (rpc.rs:206). To choose how much history a node keeps, see xOne sync modes.

Frequently asked questions

Why do some xOne log lines ignore --log.format json?

Lines written with eprintln!, such as chain:, peer refused, xone-node: and producer:, go to stderr as plain text. Only the tracing log follows --log.format.

Does disk_low mean the node stopped?

No. It means the node stopped importing blocks. The node resumes after the disk_ok line. It exits only after disk_full, with exit code 4.

Is one "setting peer aside" line a problem?

No. The node sets a peer aside for 60 s after a failed request, or for 4 s when the peer is only behind. Look for many lines with the same reason.

How do I know a producer is allowed to sign?

Look for producer: fence open: the quiet window passed, signing starts. Before that line, the producer signs nothing.

Sources