a month ago
Project: Rhino (ID 598a1984-304c-44ac-9f52-fa1c23ce7555)
Environment: production (07d3f215-7821-4c8b-a6fc-1b22862b7f1a)
Service: Postgres (ID 618105b3-a8c8-493c-a312-ec628099691d)
Data volume: postgres-volume-C4FT (ID d8f928aa-3516-47cc-a2cc-c19b88a4a357)
mount /var/lib/postgresql/data — 243 MB / 5 GB — state READYMy production Postgres is down and ALL redeploys fail at "Create container" with
NO logs produced. Railway's own auto-diagnosis says "Infrastructure Error … four
consecutive deployments failed at the same stage with no logs … image and config
are valid and were working until this date."
Timeline (UTC):
-
Until 2026-06-04 16:49: Postgres healthy (regular checkpoints, PID 82).
-
~16:49: Postgres froze — no further checkpoints, no shutdown log. App began
failing every query with "Connection terminated due to connection timeout";
/api/health returns 503.
-
19:00: last PG server log: "could not receive data from client: Connection timed out".
-
Multiple redeploys FAILED at create-container (deployment IDs: 72b180df,
d10c2bdf, fbe6b8f8, b9ac6230, 9c957028).
-
railway downremoved the frozen deployment (03077ea5, from 2026-04-25);subsequent
redeploy --from-sourcestill fails at create-container.
Request: please repair the infrastructure preventing container scheduling for this
service / migrate volume postgres-volume-C4FT (d8f928aa-3516-47cc-a2cc-c19b88a4a357)
to a healthy host. DATA MUST BE PRESERVED — do not wipe the volume.
Note: there are two unused orphan volumes in this env (postgres-volume / db838b22,
rhino-db-volume / d9083497, both detached). The active data volume is
postgres-volume-C4FT only.
3 Replies
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
Thanks for the thorough timeline and for spelling out the volume IDs — that's exactly what's needed, and your read is correct: this is a platform-side issue, not your image or config. Today's US West networking disruption left some services unable to create a container because the volume is pinned to an affected host, which matches your four consecutive create-container failures with no logs.
To be clear on the important part: your active data volume postgres-volume-C4FT (d8f928aa-3516-47cc-a2cc-c19b88a4a357) will be preserved — no wipe — and the two detached orphan volumes will be left alone. This is the case that needs platform-side recovery to migrate the service onto a healthy host; it can't be resolved from your end. We're escalating it now with the IDs you provided. I don't have a firm ETA yet but will follow up here.
Status changed to Awaiting User Response Railway • about 1 month ago
jake
Thanks for the thorough timeline and for spelling out the volume IDs — that's exactly what's needed, and your read is correct: this is a platform-side issue, not your image or config. Today's US West networking disruption left some services unable to create a container because the volume is pinned to an affected host, which matches your four consecutive create-container failures with no logs. To be clear on the important part: your active data volume **postgres-volume-C4FT** (`d8f928aa-3516-47cc-a2cc-c19b88a4a357`) will be preserved — no wipe — and the two detached orphan volumes will be left alone. This is the case that needs platform-side recovery to migrate the service onto a healthy host; it can't be resolved from your end. We're escalating it now with the IDs you provided. I don't have a firm ETA yet but will follow up here.
a month ago
Update: the service recovered on its own yesterday evening (≈23:00 UTC, 2026-06-04).
After a railway down + redeploy, a deployment finally got scheduled on a healthy
host and Postgres came back up with ALL data intact (verified: row counts match,
/api/health returns 200). So the immediate outage is resolved on my side.
Two questions:
-
Is there any residual risk of the volume (postgres-volume-C4FT,
d8f928aa-3516-47cc-a2cc-c19b88a4a357) being re-pinned to the affected US West
host? Should it be proactively migrated to a healthy host to avoid a repeat?
-
Can this be considered resolved, or do you still need anything from me?
Thanks for the quick response and for confirming the data would be preserved.
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
Glad it recovered with all data intact. Once a deployment lands on a healthy host, the volume stays there - it won't drift back to the previously affected one, so no proactive migration is needed. This is resolved on our end too, nothing else needed from you. If you haven't already, enabling volume backups on your Postgres service is worth doing for peace of mind going forward.
Status changed to Awaiting User Response Railway • about 1 month ago
Status changed to Solved dizzydes90 • about 1 month ago