xOne writes text logs by default. Pass --log.format json to get one JSON object per line. Two kinds of line do not follow --log.format: the slow-block line and the producer lines. Both go to stderr and have their own shape.

This post describes xOneRS (the Rust node) on main today. For metrics and alerts, see how to monitor an xOne node. For run steps, see node operations.

How do I choose the log format?

Pass --log.format text (the default) or --log.format json. The names terminal and logfmt also select text. Any other value is refused when the node starts.

xone --network apothem --datadir <datadir> \
     --verbosity 3 --log.format json

The tracing log goes to stdout, so a supervisor such as journald collects it. The node has no flag for a log file.

  • Text. One line is <unix ms> LEVEL target message k=v ....
  • JSON. One object per line. The fixed keys are t (unix ms), lvl, target and msg. Event fields follow as more keys. Numbers and booleans keep their type. Everything else is a string.
  • Redaction. Secrets print as <redacted> in both formats.
  • Start and stop lines. In JSON mode they are also log objects, so every stdout line parses.

Do not rely on key order in a JSON line. Read fields by name.

How do I set the verbosity?

Use --verbosity <n>. The levels are 0 silent, 1 error, 2 warn, 3 info, 4 debug and 5 detail. The default is 3.

--log.vmodule sets the level for one module. It takes target=level pairs, separated by commas. A target is a crate or module path, and the longest match wins.

xone --network apothem --datadir <datadir> \
     --verbosity 3 --log.vmodule xone_net=4,xone_sync::snap=5

Geth matches file patterns here. xOne matches module paths. -vmodule is accepted as a deprecated alias.

You can also change both at run time with the debug_verbosity and debug_vmodule RPC methods. debug_vmodule replaces the module levels. An empty string clears them.

The same settings exist as config keys under logging: verbosity, format, vmodule and slow_block.

What are the log budgets and suppression lines?

The node limits how many bytes each subsystem can write. When it drops lines, it counts them and writes one summary line.

REFERENCE TABLEScroll to explore all columns →
Subsystem Sustained Burst
consensus (xone_consensus, xone_signer) 8 KiB/s 32 KiB
exec (xone_executor, xone_evm, xone_state, xone_builder) 8 KiB/s 32 KiB
net (xone_net), sync (xone_sync), rpc (xone_rpc) 4 KiB/s 16 KiB
db (xone_db), other 2 KiB/s 8 KiB
  • Global cap. 32 KiB/s with a burst of 128 KiB.
  • Error lane. ERROR lines use their own lane, 4 KiB/s with a burst of 16 KiB. A flood of INFO lines cannot hide an error.
  • Queue. The writer queue holds 8,192 lines. When it is full, the node drops the newest line. Logging does not block hot threads.
  • Summary. At most one per subsystem every 10 s.

In text mode the summary reads suppressed lines=<n> bytes=<n> subsystem=<name>. In JSON mode it is a WARN object with msg set to suppressed and the keys lines, bytes and subsystem. The node also writes any owed summary when it stops.

To alert on drops, use the counters xone_log_suppressed_lines_total and xone_log_dropped_lines_total. The file sink and the journald rate limit are not set by the node. Under systemd, set the rate limit in the unit (ADR-007).

What does the slow-block line contain?

--debug.logslowblock <duration> writes one JSON line for each imported block that took at least that long. It is off by default (-1ns). The value 0 logs every block. A negative value turns it off.

xone --network apothem --datadir <datadir> --debug.logslowblock 500ms

The value is a Go duration, such as 500ms, 1s or 1m30s. The line has this shape:

  • level (warn) and msg (Slow block)
  • block: number, hash, gas_used, tx_count
  • timing: verify_ms, senders_ms, execution_ms, commit_ms, history_ms, total_ms
  • throughput: mgas_per_sec

This line is not a log-envelope object. It has no t, lvl or target key, and it is JSON in text mode too. The node writes it to stderr, so the budgets and --verbosity do not apply to it. Each line adds one to xone_slow_blocks_total.

With stderr and stdout merged, select it by its message:

jq -c 'select(.msg=="Slow block") | {n: .block.number, ms: .timing.total_ms}'

Which producer lines should I grep for?

Two lines show that a block producer works: sealed and sent and fence open. They start with producer:. They are plain text on stderr, also in JSON mode.

  • producer: fence open: the quiet window passed, signing starts appears once the signer fence opens. Before it, the node signs nothing. See ADR-010.
  • producer: block <n> <hash> sealed and sent (seal <ms> ms, import <ms> ms) appears for each block the node seals. The seal time covers the safety-record write and sync plus the signature, so a stalled disk shows as a large number. One rehearsal log shows seal 18 ms, import 127 ms (gate D rehearsal).
grep -E 'sealed and sent|fence open' <log-file>

For a validator, see the net2040 validator guide.

How do I ship JSON logs to a log store?

The repository docs do not say how. No guide for Loki, Elasticsearch, Datadog or a similar store is on main. The node writes JSON lines to stdout, and the slow-block and producer lines go to stderr. How you collect those streams is your choice. This post does not compare the log options of other clients, because no competitor claim was collected.

Frequently asked questions

How do I make xOne write JSON logs?

Pass --log.format json. The node then writes one JSON object per line to stdout, with the keys t, lvl, target and msg plus the event fields.

Does --log.format json change the slow-block line?

No. The slow-block line is always one JSON object on stderr, with the keys level, msg, block, timing and throughput. It is not a log-envelope object.

How do I turn on the slow-block log?

Pass --debug.logslowblock with a Go duration, such as 500ms. The default -1ns is off. The value 0 logs every imported block.

Why do I see suppressed lines= in my log?

A subsystem went over its byte budget, so the node dropped lines and counted them. It writes one summary per subsystem every 10 s. The counter xone_log_suppressed_lines_total tracks the total.

Can I change the log level without a restart?

Yes. Call the debug_verbosity RPC method for the default level and debug_vmodule for per-module levels.

Sources