To upgrade an xOne node, check the new binary, stop the unit, back up the whole datadir, swap the binary and start the unit again. To roll back, put the old binary back, or restore the backup if the datadir format changed. Never lower or restore the signer safety record.
How is an xOne node installed?
The release unpacks to /opt/xone. A systemd template unit, xone@<net>, starts one node per network. The launcher reads DATADIR and ETHSTATS_SECRET from /etc/xone/<net>.env.
/opt/xone/{xone,run-xone.sh,[email protected],bootnodes/}is the layout.systemctl stopsends SIGTERM. The node exits in 5 s or less, and systemd sends SIGKILL after 15 s.- Exit code 0 is a clean stop. Code 3 means a service did not stop in time.
- A
.debor.rpmpackage keeps/etc/xoneand the datadir. It never starts the unit.
How do I check a new xOne binary?
Check SHA256SUMS and the version stamp in a scratch directory before you stop anything.
gpg --verify SHA256SUMS.sig SHA256SUMS && sha256sum -c SHA256SUMS
/opt/xone/xone --version
The stamp is xOneRS/v<semver>-<commit8>-<DDMMYYYY>/<os>-<arch>/rustc<version>. It must have no unknown and no -dirty. See how the stamp works.
Two limits apply today. The v0.1.0 release page lists xone-v0.1.0-linux-amd64 and SHA256SUMS only. And no release key exists yet: the release document calls it an open owner decision. The signature step needs a key from the owner.
Which hosts do I upgrade first?
Followers go first. A producer host runs a tag only after a follower has run it. Then you upgrade producer hosts one at a time, with the owner present.
Each of a net2040 follower and an Apothem follower must pass this soak:
- 24 hours or more, across at least one epoch switch.
- 0 refused blocks,
eth_syncingfalse at the end, and the head within 10 minutes of the tip. xone --versionequals the tag, with no restart that the operator did not start.
What happens to the datadir format?
An older binary always refuses a newer datadir. There is no in-place downgrade. The runbook tells you to read the "datadir format" line of the release notes first.
- Format unchanged: roll back by installing the old binary and starting it.
- Format changed: roll back by restoring the backup you took before the upgrade. Without a backup, only a resync works.
The v0.1.0 changelog has no such line. ADR-005 says the datadir manifest has one version, and the binary refuses another with FormatVersion. ADR-005 describes xone db migrate as a future command, and the CLI list does not include it. One migration does exist on main, after v0.1.0: a state-table schema 1 datadir moves to schema 2 on start. A build that finds a higher schema refuses to open ("Use the build that wrote it"). Take the backup before you start such a build.
How do I upgrade and roll back?
Stop the node, copy the datadir, swap the binary, start, and prove. The backup needs the same free space as the datadir.
systemctl stop xone@<net> # then: systemctl is-active xone@<net>
cp -a <datadir> <datadir>.pre-<N+1>
mv /opt/xone /opt/xone.N && install -d /opt/xone && cp -r xone-<N+1>-linux-x86_64/. /opt/xone/
/opt/xone/xone --version # the stamp of N+1
systemctl start xone@<net>
journalctl -u xone@<net> -n 20 # no FormatVersion, no refused datadir
The head must rise, and web3_clientVersion must equal the new stamp. With a package, use apt install ./xone_<N+1>_amd64.deb, start the unit yourself, and roll back with apt install xone=<N>.
Rollback, format unchanged:
systemctl stop xone@<net>
rm -r /opt/xone && mv /opt/xone.N /opt/xone && systemctl start xone@<net>
Rollback, format changed:
mv <datadir> <datadir>.failed && cp -a <datadir>.pre-<N+1> <datadir>
Then put the current safety record back (next section) and restore the old binary. Keep /opt/xone.N and the backup until N+1 has run one epoch. If the head does not rise after 5 minutes with peers connected, roll back and keep the failed datadir and the journal.
Why must I never lower the signer safety record?
The safety record stops a signer from signing two different things in one round. It only rises. An older record can allow a double sign.
- Do not delete it, restore it from a backup or copy it between hosts.
- After a chain rollback the signer may refuse to vote in rounds it visits again. That is the intended cost.
- The node cannot tell a restored record from a current one. It treats the next start as unproven.
- Every signer start waits a quiet window before it signs: at least 120 s on mainnet, Apothem and net2040. An upgrade of a producer costs that window.
The record lives at <datadir>/xone/safety. The runbook and ADR-010 now name that path; issue #820, which tracked an older datadir/safety wording, was closed on 2026-11-14. A copy of the whole datadir includes the record. Step B3 of the runbook now uses ops/tools/xone-rollback-restore.sh, which keeps the current record, keeps the failed datadir, and stops before the swap if a safety file is missing or fails to copy. The ADR-010 status is still "proposed". A file lock refuses a second process on one datadir. It does not stop a second host.
What did the rehearsal cover?
The recorded rehearsal, dated 2026-11-12, covered the upgrade, one rollback with an unchanged format, and a restore of the pre-upgrade copy. It ran on a loopback devnet and passed.
- Each start printed a
web3_clientVersionequal toxone --version. The head block hash equalled geth's. - Each stop took about 0.1 s with exit code 0. Each restart resumed at block 17.
- It did not cover a format migration, a signer,
systemctl startandstop, or a public network. - The 15-minute install by a second person is still to do (issue #46).
Do the other three clients have upgrade steps?
No. The xOneTS, xOnePy and xOneC release pages are mirrored from the Rust rulebook. They end with a SHA256SUMS file and a check that the node reports the same stamp on SkyNet. They have no upgrade or rollback steps. Their repositories are private. See which xOne client to use and node operations.
How do other clients describe upgrades?
| Client | What the project says | xOneRS |
|---|---|---|
| ethrex | Upgrades are a drop-in restart. As stated in Six months of ethrex in production, 2026-07-20. | Stop, back up, swap, start. A newer datadir under an older binary is refused. |
Frequently asked questions
Can I downgrade an xOne node in place?
No. An older binary always refuses a newer datadir. If the format changed, restore the backup taken before the upgrade.
What do I back up before an upgrade?
Stop the node, then copy the whole datadir with cp -a. The copy needs the same free space as the datadir.
Can I restore the safety record from a backup?
No. The record only rises. Restoring an older one is outside the node's control, and the node treats the next start as unproven.
Sources
- xOneRS: ops/upgrade-rollback.md, ops/install.md, ops/README.md, ops/[email protected]
- xOneRS: docs/ops/release.md, docs/ops/release-rehearsal.md, docs/ops/keys.md, docs/cli/flags.md
- xOneRS: ADR-005, ADR-010, reconciliation note, producer.rs, safety_file.rs, CHANGELOG.md, v0.1.0 release
- xOneTS (private): docs/process/release.md. xOnePy (private): docs/process/release.md. xOneC (private): docs/process/release.md
This article describes the source and evidence at its review date. Check the linked repository for current implementation and qualification status.
Explore the engineering evidenceEdit this page on GitHub opens the Markdown source (content/blog/upgrade-rollback-howto/index.md). GitHub may ask you to sign in. Report a correction opens a prefilled GitHub issue for this page.