2 months ago
Since Aug 10, our backend (Node/Express + pg pool, service: gom-backend) has been experiencing recurring, intermittent Postgres connection failures across many unrelated routes, causing real 500 errors visible to end users.
Error pattern (repeated across multiple bursts throughout the day):
- "Connection terminated unexpectedly"
- "Connection terminated due to connection timeout"
- Originating from node-postgres (pg / pg-pool) at pg-pool/index.js:45:11
Affected routes (all failing independently, at different timestamps):
- GET /api/listings
- GET /api/listings/stats
- GET /api/features/reports
- GET /api/features/website-reviews/public
- GET /api/features/support/unread
- GET /api/messages/unread/count
- GET /api/auth/profile/stats
- GET /api/services/box/availability
- POST /socket.io/
Timestamps observed (UTC, this is not exhaustive — just what we captured in Sentry):
- ~03:14–03:17 AM (cluster shortly after a service restart)
- ~11:16 AM
- ~11:26 AM
Client-side impact confirmed via browser DevTools: these backend 500s also surface as CORS-policy-blocked errors in the browser (since the response fails before CORS headers are attached), causing listings, stats, and unread counts to visibly fail to load for real users during these windows.
Plan: Hobby. Usage this billing cycle is minimal ($0.09 of $5.00), so this doesn't appear to be credit/throttling related.
We already added a pool.on('error') handler on our end to prevent an idle-client error from crashing the whole process, but the underlying connection timeouts to Postgres are still occurring. Could you check whether there were any connectivity issues, restarts, or instability on the Postgres side of this project during the timestamps above?
Happy to share full Sentry traces or Railway service/project IDs if helpful.
16 Replies
2 months ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 2 months ago
2 months ago
I'm seeing the same thing. Since Railway automatically applying a CVE patch, my Postgres DB has been unresponsive. I tried redeploying my DB but it's stuck at "creating containers" for >6 minutes. When I go to my DB backups tab, the content isn't loading. Please help, Railway!
trevorblades
I'm seeing the same thing. Since Railway automatically applying a CVE patch, my Postgres DB has been unresponsive. I tried redeploying my DB but it's stuck at "creating containers" for >6 minutes. When I go to my DB backups tab, the content isn't loading. Please help, Railway!
2 months ago
FWIW, my DB is deployed in US West
2 months ago
same here. on US west. my pages are loading slow intermittently. sometimes not at all. this is on prod and staging.
2 months ago
US West currently has issues, and the Team is aware of it. I'd try deploying the service in a different region for now.
2 months ago
I tried deploying my service to US East and it's been migrating the volume for the last 22 minutes. Is this normal? My volume is 3GB in size.
2 months ago
The migration might be affected by the incident as well, I wouldn't recommend making such changes to stateful US-West services during this incident. Don't worry, your data likely isn't lost, Railway makes a backup before performing any migrations, just don't attempt to delete the volume or anything!
0x5b62656e5d
Is the service deployed in US West?
2 months ago
Yeah I solved it by changing the region to southeast asia because most of my users are based there so far it works
dev
The migration might be affected by the incident as well, I wouldn't recommend making such changes to stateful US-West services during this incident. Don't worry, your data likely isn't lost, Railway makes a backup before performing any migrations, just don't attempt to delete the volume or anything!
2 months ago
Appreciate the heads up! Actually the migration seems to have helped a bit. I was getting connection timeout errors specifically on US West, right around when this incident started. Since moving to Southeast Asia it's been a lot quieter.
Also most of my users are in the Philippines anyway so SEA was probably the better region for me long term regardless of the incident.
Data came through fine, nothing lost. Thanks for confirming the backup thing though, good to know.
dev
The migration might be affected by the incident as well, I wouldn't recommend making such changes to stateful US-West services during this incident. Don't worry, your data likely isn't lost, Railway makes a backup before performing any migrations, just don't attempt to delete the volume or anything!
2 months ago
I'm getting mixed messages here. 0x5b62656e5d just said "I'd try deploying the service in a different region" and you're saying "I wouldn't recommend making such changes." That's super confusing...
Good to know that backups have been made. I'll sit tight and wait for a resolution.
trevorblades
I'm getting mixed messages here. 0x5b62656e5d just said "I'd try deploying the service in a different region" and you're saying "I wouldn't recommend making such changes." That's super confusing... Good to know that backups have been made. I'll sit tight and wait for a resolution.
2 months ago
I believe 0x5b62656e5d was referring to services in general, I'm specifically talking about stateful services (those with volumes attached). That's just my opinion though, you're welcome to migrate them over, but I'm seeing a few reports that migrations are getting stuck which is worth considering before migrating
Appreciate the heads up! Actually the migration seems to have helped a bit. I was getting connection timeout errors specifically on US West, right around when this incident started. Since moving to Southeast Asia it's been a lot quieter. Also most of my users are in the Philippines anyway so SEA was probably the better region for me long term regardless of the incident. Data came through fine, nothing lost. Thanks for confirming the backup thing though, good to know.
2 months ago
Glad to hear migrating helped!
dev
I believe 0x5b62656e5d was referring to services in general, I'm specifically talking about stateful services (those with volumes attached). That's just my opinion though, you're welcome to migrate them over, but I'm seeing a few reports that migrations are getting stuck which is worth considering before migrating
2 months ago
Dang, that sucks. My migration definitely seems like it's stuck. Any way to roll it back or stop the process and restore the backup?
trevorblades
Dang, that sucks. My migration definitely seems like it's stuck. Any way to roll it back or stop the process and restore the backup?
2 months ago
Unfortunately no way to do that from the UI that I know of, if it's still stuck once the incident is over then you can make another help thread so a team member may assist you
Status changed to Awaiting User Response Railway • about 2 months ago
fvr1
I also did a the DB migration to US East, and it's stuck… what should I do?
2 months ago
If it's still stuck by the time the incident is over then I'd recommend opening another help thread
Status changed to Solved Anonymous • about 2 months ago