a month ago
Project ID: a4836f78-3eca-4806-880f-9b27daa99abe
Affected services: Backend "Allerik - Backend" (ee8117a9-5267-475f-a4ff-7141939876b8) & Postgres-l5bk (9b6271a7-4042-4376-91df-1d7946e47cdf)
Duration: 7h+ since 05:40:03 UTC (Postgres crashed unsolicited) → recovered 11:33:58 UTC → still broken.
Root cause: backend on host metal-004-02 cannot reach Postgres on host aws-us-west-1a-r8i12xlarge-76. Tested and ALL fail: internal hostname postgres-l5bk.railway.internal:5432, raw private IPv6 [fd12:2415:413e:1:5000:ac:5df6:be88]:5432 (bypasses DNS), and public proxy gondola.proxy.rlwy.net:14543 (sslmode default/require/disable). DB is healthy — SELECT 1 works from the query console; pg_hba is host all all all scram-sha-256; listening on *:5432. Postgres logs show connections resetting before auth. This is a cross-host private-network/egress failure on the backend service. Please repair the mesh routing or reschedule the backend to a healthy host.
3 Replies
a month ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 1 month ago
a month ago
Try to redeploy source image by clicking on your postgres service, then press CTRL+K > redeploy source image. Also, make sure your backend and database are in the same region. If redeploying doesn't help, I'd try to change the db's region temporarily to rule out infra issues.
a month ago
Update — I followed the region advice, and it's now confirmed this is NOT a region/DNS/config issue. It's platform-level private networking.
What I did since posting:
- Migrated the Postgres to Metal us-east4 (volume moved cleanly; DB healthy — SELECT 1 still returns from the query console).
- Moved the backend to the SAME region — both now on Metal US East (Virginia).
- Redeployed the backend pointing at the internal hostname postgres-l5bk.railway.internal:5432 → still "DB not ready", 30/30 attempts, container exits and loops.
- As a decisive test, I pointed the backend's DATABASE_URL directly at the DB's CURRENT raw private IPv6 [fd12:2415:413e:1:8000:e0:41c2:ba67]:5432, bypassing DNS entirely → STILL cannot connect, looping "DB not ready" for over an hour.
So: two services in the SAME Metal region, connecting by RAW private IPv6, and the backend still can't reach the Postgres. This rules out DNS, region mismatch, SSL (pg_hba is host all all all scram-sha-256), and the DB being down. This is a Railway private-networking failure at the platform level — nothing on my side can fix it.
Can a Railway engineer please inspect the private-network mesh for project a4836f78-3eca-4806-880f-9b27daa99abe? Production has been down ~20 hours. Services: backend ee8117a9-5267-475f-a4ff-7141939876b8 and Postgres 9b6271a7-4042-4376-91df-1d7946e47cdf.
a month ago
Closed!
Status changed to Solved raraujo0889 • about 1 month ago