Skip to main content

​
Overview

Rivestack HA clusters use Patroni for automatic failover and cluster management. When a primary node fails, an eligible replica can be promoted automatically; applications should expect a brief interruption and reconnect.
Automatic failover requires an HA cluster (2+ nodes). The Free and Solo plans run a single node and do not include failover. Solo is a good fit when you want dedicated, backed-up PostgreSQL but don’t yet need multi-node resilience.

​
Architecture

Each HA cluster consists of:
  • 1-3 PostgreSQL nodes: one primary and zero or more streaming replicas
  • 1 Hetzner Load Balancer for clusters with two or more nodes
  • HAProxy on each database node: receives PROXY protocol, applies the client IP allowlist, and passes TLS through to PgBouncer
  • etcd: distributed consensus store for leader election
  • Patroni: cluster manager that handles replication, failover, and health checks

​
How failover works

  1. Detection: Patroni continuously monitors the health of each PostgreSQL node. If the primary becomes unresponsive, Patroni detects the failure within seconds.
  2. Leader election: Patroni uses etcd to coordinate leader election. An eligible replica is selected as the new primary.
  3. Promotion: The selected replica is promoted to primary. It begins accepting writes immediately.
  4. Routing: The load balancer detects the topology change and routes new connections to the new primary. Existing connections to the old primary are terminated.
  5. Recovery: When the failed node comes back online, it automatically rejoins the cluster as a replica and begins streaming from the new primary.

​
Failover expectations

Rivestack does not currently promise a zero RPO or a fixed failover SLA. If your workload requires synchronous commit or a contractual recovery objective, contact us before relying on automatic failover.
During failover, active connections to the old primary will be dropped. Your application should implement connection retry logic to handle brief interruptions.

​
Replication

Rivestack uses PostgreSQL streaming replication to keep replicas in sync with the primary:
  • WAL (Write-Ahead Log) records are streamed from primary to replicas in real time.
  • Replicas are used for automatic failover.
  • Replication lag is monitored and visible in the dashboard Metrics tab.

​
Replication lag

Monitor replication lag for each replica in the dashboard:
  • Lag time: How far behind the replica is (in seconds)
  • Lag bytes: Size of un-replayed WAL data
Replication lag varies with write throughput, node load, and network conditions. Monitor it for your workload; do not assume it is always sub-second.

​
Node roles

​
Connection routing

All connections use port 6432 — the same port a single-node cluster uses, so scaling to HA leaves your connection string untouched. The load balancer forwards only to the node whose Patroni /primary health check is healthy. Applications must reconnect after a failover.

​
Best practices

During failover, connections to the old primary are terminated. Your application should:
  • Catch connection errors and retry after a short delay (1-3 seconds).
  • Use exponential backoff for retries.
  • Most PostgreSQL drivers handle reconnection automatically if configured.
A single-node cluster has no failover target. Run at least 2 nodes in production to ensure automatic recovery from node failures.
Check the Metrics tab regularly. Sustained replication lag may indicate the replica needs more resources or there are network issues.
Verify your application handles failover gracefully by observing behavior during planned maintenance windows.