4 months ago
Our Postgres HA cluster experienced a critical failure where the primary node (postgres-1) became unable to start due to corrupted WAL (Write-Ahead Log) files. This created a deadlock:
postgres-1 was stuck in recovery and couldn't start
postgres-2 & postgres-3 were healthy but couldn't proceed without a primary
Patroni (the HA orchestrator) couldn't failover because postgres-1 held the leader lock
RollCall app crashed on startup trying to connect during database migrations
The primary node cycled between appearing healthy (health checks passed) and unavailable (503 errors), never stabilizing.
My live site is down two days now. My staging has no issues. Production is totally dead.
1 Replies
4 months ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 4 months ago
3 months ago
What DCS is backing Patroni — etcd, Consul, or Kubernetes? (changes how we force-expire the lock if step 2 doesn't clear it)
Output of patronictl list?