2 months ago
Our production Postgres service in us-west2 now shows SUCCESS/RUNNING, but it is still unreachable. Direct connections terminate, port 5432 has no listener, and the Website remains degraded. After the incident started, we created one backup and it shows <1 MB in the UI, while the CLI reports referencedMB: null and usedMB: null. Recently without any actions was created a few more empty backups. Could Railway confirm whether these new incremental backups completed successfully and contain the full latest database state? Should we restore the newest backup, or wait for the regional incident to be fully resolved?
5 Replies
2 months ago
We're sorry for the trouble. This is caused by an ongoing incident, Connectivity issues in US West, affecting networking and deployments in the US West region. Your Postgres service being unreachable despite showing as running, and the backup anomalies (near-zero size, null values), are consistent with this. We'd recommend waiting for the incident to be fully resolved before acting on any backup taken during the disruption, as connectivity issues can prevent backups from completing properly. Follow updates at the incident link above.
Status changed to Awaiting User Response Railway • about 2 months ago
2 months ago
The connection is still not working. Is it safe to redeploy or restart Postgres?
Attachments
Status changed to Awaiting Railway Response Railway • about 2 months ago
2 months ago
Apologies for the trouble. The Connectivity issues in US West incident has since been resolved, and the resolution notes confirm that a small number of services may need a redeploy to fully recover. It is safe to redeploy your Postgres service now. We would recommend taking a fresh backup after the redeploy succeeds and connectivity is confirmed, rather than relying on backups created during the disruption.
Status changed to Awaiting User Response Railway • about 2 months ago
zimoby
The connection is still not working. Is it safe to redeploy or restart Postgres? 
2 months ago
Try redeploying - now that the incident is over, redeploying should help and allow you to reconnect
Status changed to Awaiting Railway Response Railway • about 2 months ago
Status changed to Awaiting User Response Railway • about 2 months ago
milo
Try redeploying - now that the incident is over, redeploying should help and allow you to reconnect
2 months ago
yes working, thanks!
Status changed to Awaiting Railway Response Railway • about 2 months ago
Status changed to Solved Railway • about 2 months ago