Skip to main content

​
Overview

Rivestack HA clusters use Patroni for automatic failover and cluster management. When a primary node fails, a replica is promoted within seconds — no manual intervention required.

​
Architecture

Each HA cluster consists of:
  • 1-3 PostgreSQL nodes — one primary and zero or more replicas for failover
  • 2 load balancers — route connections to the correct node with a floating VIP (Virtual IP)
  • etcd — distributed consensus store for leader election
  • Patroni — cluster manager that handles replication, failover, and health checks
                    ┌──────────────┐
   Clients ────────►│Load Balancer │
                    │  (VIP: 5432) │
                    └──────┬───────┘
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
         ┌─────────┐ ┌─────────┐ ┌─────────┐
         │ Primary  │ │ Replica │ │ Replica │
         │ (node 1) │ │ (node 2)│ │ (node 3)│
         └─────────┘ └─────────┘ └─────────┘
              │            ▲            ▲
              └──── streaming replication──┘

​
How failover works

  1. Detection — Patroni continuously monitors the health of each PostgreSQL node. If the primary becomes unresponsive, Patroni detects the failure within seconds.
  2. Leader election — Patroni uses etcd to coordinate leader election. The healthiest replica with the least replication lag is selected as the new primary.
  3. Promotion — The selected replica is promoted to primary. It begins accepting writes immediately.
  4. Routing — The load balancer detects the topology change and routes new connections to the new primary. Existing connections to the old primary are terminated.
  5. Recovery — When the failed node comes back online, it automatically rejoins the cluster as a replica and begins streaming from the new primary.

​
Failover expectations

MetricValue
Detection time~5-10 seconds
Promotion time~5-15 seconds
Total failover time~10-30 seconds
Data lossZero (synchronous commit with streaming replication)
During failover, active connections to the old primary will be dropped. Your application should implement connection retry logic to handle brief interruptions.

​
Replication

Rivestack uses PostgreSQL streaming replication to keep replicas in sync with the primary:
  • WAL (Write-Ahead Log) records are streamed from primary to replicas in real time.
  • Replicas are used for automatic failover.
  • Replication lag is monitored and visible in the dashboard Metrics tab.

​
Replication lag

Monitor replication lag for each replica in the dashboard:
  • Lag time — How far behind the replica is (in seconds)
  • Lag bytes — Size of un-replayed WAL data
Under normal conditions, replication lag is sub-second. High write throughput or network issues can temporarily increase lag.

​
Node roles

RoleReadsWritesCount
PrimaryYesYesExactly 1
ReplicaFailover onlyNo0-2

​
Connection routing

All connections go through port 5432. The load balancer automatically routes traffic to the current primary and handles failover transparently — no configuration needed.

​
Best practices

During failover, connections to the old primary are terminated. Your application should:
  • Catch connection errors and retry after a short delay (1-3 seconds).
  • Use exponential backoff for retries.
  • Most PostgreSQL drivers handle reconnection automatically if configured.
A single-node cluster has no failover target. Run at least 2 nodes in production to ensure automatic recovery from node failures.
Check the Metrics tab regularly. Sustained replication lag may indicate the replica needs more resources or there are network issues.
Verify your application handles failover gracefully by observing behavior during planned maintenance windows.