18 days ago
Postgres healthy but completely unreachable — restart + redeploy don't fix (production down)
Railway-hosted Postgres (postgres-ssl:17), production. The DB container is healthy (deployment SUCCESS, normal checkpoint logs continuing through ~06-18 05:27 UTC, no PANIC/FATAL/OOM/disk-full — but no client can connect on any path:
- App → *.railway.internal:5432 (private network): Prisma P1001: Can't reach database server → crash-looping.
- External → public TCP proxy (endpoint omitted): nc opens the TCP socket, but psql returns "server closed the connection unexpectedly" the instant the Postgres protocol starts — across all SSL modes (require/prefer/disable) and gssencmode=disable.
Already tried (no effect): restarted the Postgres service, then redeployed it. Redeploying doesn't work for the same reason - still 'deploying' after 9 minutes.
Suspected: the proxy/private-network attachment for this service is broken — container is up and writing checkpoints, but neither the internal IPv6 private network nor the public TCP proxy completes a connection.
Logs:
LOG: received direct SSL connection request without ALPN protocol negotiation extension
LOG: could not accept SSL connection: no application protocol
LOG: could not accept SSL connection: unsupported protocol
HINT: client does not support any SSL protocol version between TLSv1.2 and TLSv1.3
LOG: could not accept SSL connection: no shared cipher / bad key share
8 Replies
Status changed to Awaiting Railway Response Railway • 18 days ago
18 days ago
@railway any update?
Status changed to Awaiting Railway Response Railway • 18 days ago
18 days ago
Guys the DB has been 'Deploying' for 12 minutes now - and I am waiting almost 40 mins for a response. The whole site is down. Please....
18 days ago
i've the same problem here @railway
18 days ago
I am trying to get railway's attention on discord....prod is completely down and waiting more than an hour for a response....customers can't access any of their data, not sure how it's possible to build a real product on railway when support doesn't respond
Other railway projects seem ok, but that's not much consolation right now.
18 days ago
Seems to be back now. An email went out 1 hour after I reported (reported here at 8:50 CET, email sent out at 9:53 CET) to say the service was down, but I have no idea why no one from support replied here to say they were looking into it in the meantime...
Honestly support for the EU in the morning is very thin on the ground. We lost a lot of revenue due to the server being down for 90 mins.
17 days ago
We had a host hardware failure on June 19 that took your Postgres offline and left the redeploy unable to complete until the host recovered, which is why it sat in Deploying. Your database and shipping API came back at 08:00 UTC and have been running normally since. Apologies for the downtime, and for the slow response during EU morning hours — we know how costly an outage like this is, and we're following up internally on the response gap. Can you confirm everything's healthy on your end?
Status changed to Awaiting User Response Railway • 17 days ago
12 days ago
Yes it's healthy now, but we lost around 500 EUR in sales due to the 90 minute downtime and customers could not access their profiles and get delivery updates, leading to missed pickups. It's essential to have better support in EU working hours.
Status changed to Awaiting Railway Response Railway • 12 days ago
12 days ago
Glad it's back. We're sorry for the downtime and for the slow response while your production was down. That's not acceptable and your frustration is completely warranted. We're aware EU morning coverage needs to improve.
Status changed to Awaiting User Response Railway • 12 days ago
Status changed to Solved sam-a • 12 days ago