How I upgrade blockchain nodes with near-zero downtime in production

There’s nothing worse than waking up to find your node lagging behind…..not because it crashed, but because you hit “upgrade” at the wrong time.

Most blockchain node outages don’t happen during sync.

They happen during upgrades.

Client releases drop, teams rush to upgrade, traffic breaks, data locks, or worse, corrupted state.

Here’s the boring but reliable way we handled node upgrades in production, repeatedly, with minimal downtime.

Before we proceed

My approach assumes: You are running more than one node instance behind a load balancer.

If you are running a single node in production, this is your sign to stop reading and fix that first(for real).

The setup

My setup looked like this:

• 2–3 blockchain node instances

• All behind an AWS Load Balancer

• Traffic only goes to healthy nodes

Each node has:

• Its own local data directory

• Independent health checks

The real problem upgrades cause

Upgrading a node is not always just:

“Download new binary and restart”

It might involve disk migrations, database rewrites (for example, LevelDB to PebbleDB on Geth), state compatibility checks, or unpredictable startup times.

In other words, an “upgrade” is often a controlled failure plus a recovery plan. The only question is whether you spread that failure across nodes, or concentrate it on all of them at once.

The solution I did

I have 2 scripts.

First script

Checks the client’s GitHub repo tags and compares them with the version currently running on the node.

If a newer release exists, it sends an event to my backend DB (I used Timestream, but any DB works) and triggers a Grafana alert that notifies me on Slack.

Once I receive the alert, I decide when to trigger the upgrade process.

Second script

This script carries out the upgrade process for each node behind the load balancer:

1. Drain traffic from the node (via LB health check)

2. Stop the node process

3. Upgrade the client binary

4. Start the node

5. Wait until:

• Node is fully synced

• RPC is responsive

• Health checks pass

Only then…it moves to the next node

This script has two trigger modes.

The first is manual, based on a decision from the first script, because some upgrades introduce deprecated or new flags required for the node to start successfully. For example, when upgrading my Arbitrum Nitro node from v3.6.5 to v3.6.6, I had to include a new startup flag.

The second is automatic. If a node panics and remains unhealthy for more than 5–10 minutes, the script automatically upgrades the client and restarts the node. Around 95% of the time, this fixes the issue. As for the other 5%, well thats a story for another day.

Why did I consider this approach?

This approach

• Avoids full outages.

• Avoids rushed rollbacks.

• Avoids corrupted state.

• Let's you observe new client behavior safely.

• Gives you time to react if something looks off.

This matters even more during

• Hard forks

• Client-breaking releases

• Performance-heavy upgrades

• New pruning or DB formats

In conclusion, my setup isn’t clever. It’s just disciplined. If you run blockchain nodes in production and don’t have rolling upgrades, you’re one bad release away from downtime.