17 days ago
Our Postgres HA cluster experienced a critical failure where the primary node (postgres-1) became unable to start due to corrupted WAL (Write-Ahead Log) files. This created a deadlock:
postgres-1 was stuck in recovery and couldn't start
postgres-2 & postgres-3 were healthy but couldn't proceed without a primary
Patroni (the HA orchestrator) couldn't failover because postgres-1 held the leader lock
RollCall app crashed on startup trying to connect during database migrations
The primary node cycled between appearing healthy (health checks passed) and unavailable (503 errors), never stabilizing.
My live site is down two days now. My staging has no issues. Production is totally dead.
1 Replies
17 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 17 days ago
15 days ago
What DCS is backing Patroni — etcd, Consul, or Kubernetes? (changes how we force-expire the lock if step 2 doesn't clear it)
Output of patronictl list?