Update: still down, and we have now tried the self-service recovery paths we can safely try.

Current status:

• Production health check is still returning 502

• backend_database latest deployment is still:

  2e6ce1cb-1bf4-4dfa-bbbe-bc4e57dd03f0

• That deployment appears stuck in the Railway UI at:

  “Deployment in progress: Creating containers…”

• The UI shows no build logs and no deploy logs for the DB deployment.

• Postgres logs have no fresh output since 2026-06-19T13:17Z.

• PgBouncer continues to fail because it cannot reach the Postgres service:

  “Backend backenddatabase.railway.internal not accessible”

  and repeated:

  “timeout reached before the port went into state "inuse"”

• Direct Railway DB connection still fails:

  railway connect -e production backend_database

  psql: connection to server at "centerbeam.proxy.rlwy.net", port 10304 failed: server closed the connection unexpectedly

What we tried:

• Restarted services.
• Aborted the stuck DB redeploy.
• Retested direct DB connectivity.
• Redeployed backend_database again.
• Confirmed backend_app/backend_admin are failing downstream because Prisma cannot reach pgbouncer.railway.internal:6432.
• We have not detached/deleted/recreated any volumes.

Important observation:

The prior known-good backend_database deployment is still shown as ACTIVE in the Railway UI, but the newer redeploy is stuck in “Creating containers…”. We need Railway to inspect/recover the backend_database deployment/container scheduling/volume attachment/private networking path.

Can someone from Railway please look at backend_database service 45bfdd8c-fe78-4fb6-a9fa-e15d3e829842 in production environment 7396af74-073b-495e-b542-c177f0b68e4d, especially deployment 2e6ce1cb-1bf4-4dfa-bbbe-bc4e57dd03f0?

Update as of 2026-06-19T14:46Z:

Production is still down.

Current checks:

• https://v2.escargot.com/health still returns 502.

• Direct DB connection still fails:

  railway connect -e production backend_database

  psql: error: connection to server at "centerbeam.proxy.rlwy.net" (66.33.22.245), port 10304 failed: server closed the connection unexpectedly

  This probably means the server terminated abnormally before or while processing the request.

Since the original report:

• We aborted the stuck backend_database redeploy.
• We tried a fresh backend_database redeploy.
• That redeploy became stuck at “Creating containers…” with no build/deploy logs and no fresh Postgres logs.
• We then rolled back to an older backend_database deployment.

Current backend_database state:

• service ID: 45bfdd8c-fe78-4fb6-a9fa-e15d3e829842
• instance ID: 3089ae96-500c-462e-99c9-78d25f81362f
• current deployment ID: 01f3619c-f9af-4a94-811f-e9b972a9c7c5
• image: ghcr.io/railwayapp-templates/postgres-ssl:16
• environment ID: 7396af74-073b-495e-b542-c177f0b68e4d

Even after rollback, the DB is still not reachable. PgBouncer continues to report that it cannot reach backenddatabase.railway.internal and times out waiting for the backend port to become in-use. The database logs still do not show fresh Postgres startup/readiness output after the outage began.

Even the Railway UI cannot connect:

Same issue over here; multiple reports across central station, discord and x.com and no response from railway anywhere

Same incident here — production Postgres in US East, project luminous-renewal. Down since 13:24 UTC.

Identical signature: Prisma P1001 "Can't reach database server" on the private network (postgres.railway.internal), and the public proxy returns "connection terminated unexpectedly." The Postgres checkpointer still logs but the server accepts no client connections.

Restart + a clean, uninterrupted redeploy are both stuck on "Creating containers" — it can't schedule a container. Looks like the wedged container hasn't released its single-attach volume, so the new one can't mount it. +1 to force-terminating the stuck container / releasing the volume.

That's now multiple customers across Station, Discord, and X with no Railway response for 3+ hours. Happy to DM deployment IDs (stuck redeploy: 23a2f5c0).

Same issue here. Not resolving.

Stuck here.

Quick update: the host/hardware failure behind this has been mitigated, and databases that were wedged have been rescheduling onto healthy hosts. The escargot production database is back up on a fresh deployment and serving normally, with data intact (the 0 MB size some of you saw was a dashboard reporting glitch, not data loss). If your database in this thread is still showing stale connection errors, please redeploy it once so it reconnects through the recovered host, then reply here. If any of you are still fully down after a redeploy, reply with your project/service and we'll jump on that specific one immediately. Apologies for the disruption.

Sorry for the long silence on this. Confirming you're recovered: backend_database came back at 15:12 UTC yesterday and is healthy, and your data is fully intact (the volume is READY at ~9 GB; the 0B you saw was the wedged container not reporting, not data loss). Root cause was a host failure: the old container hadn't released the single-attach volume, so redeploys hung on "Creating containers" until it rescheduled onto a healthy host. We're hardening that auto-recovery path on our side. For a production database, running HA Postgres (replicas across nodes) would keep you up through a single host failure, and I'm happy to help you set that up. To everyone else who piled onto this thread: you should be recovered too; if your DB is still down, reply with your service ID and we'll get it rescheduled. — Angelo

Angelo - thanks for the response, I would appreciate help getting that set up. Let me know the best way to take this forward.

Glad you're back up. To confirm, backend_database is on a fresh healthy deployment, the volume is READY at ~9 GB, and your data is intact.

For HA Postgres, the path is to deploy our Postgres HA template (a primary plus replicas with automatic failover via repmgr), then cut your app over to it. High level:

1. Deploy the "Postgres HA" template into your production environment (Create > Template, search Postgres HA).
2. Restore your current data into the new cluster (pg_dump from backend_database, pg_restore into the HA primary). I can walk you through the exact commands, or we can do a logical replication cutover to minimize downtime.
3. Point pgbouncer / backend_app at the HA connection string and redeploy.

This keeps you serving through a single host failure, which is exactly what hit you here. If you tell me your preferred maintenance window and whether you want a dump/restore (simple, brief downtime) or a replicated cutover (near zero downtime, more steps), I'll give you the precise runbook for your services. — Angelo