11 days ago
Since Aug 10, our backend (Node/Express + pg pool, service: gom-backend) has been experiencing recurring, intermittent Postgres connection failures across many unrelated routes, causing real 500 errors visible to end users.
Error pattern (repeated across multiple bursts throughout the day):
- "Connection terminated unexpectedly"
- "Connection terminated due to connection timeout"
- Originating from node-postgres (pg / pg-pool) at pg-pool/index.js:45:11
Affected routes (all failing independently, at different timestamps):
- GET /api/listings
- GET /api/listings/stats
- GET /api/features/reports
- GET /api/features/website-reviews/public
- GET /api/features/support/unread
- GET /api/messages/unread/count
- GET /api/auth/profile/stats
- GET /api/services/box/availability
- POST /socket.io/
Timestamps observed (UTC, this is not exhaustive — just what we captured in Sentry):
- ~03:14–03:17 AM (cluster shortly after a service restart)
- ~11:16 AM
- ~11:26 AM
Client-side impact confirmed via browser DevTools: these backend 500s also surface as CORS-policy-blocked errors in the browser (since the response fails before CORS headers are attached), causing listings, stats, and unread counts to visibly fail to load for real users during these windows.
Plan: Hobby. Usage this billing cycle is minimal ($0.09 of $5.00), so this doesn't appear to be credit/throttling related.
We already added a pool.on('error') handler on our end to prevent an idle-client error from crashing the whole process, but the underlying connection timeouts to Postgres are still occurring. Could you check whether there were any connectivity issues, restarts, or instability on the Postgres side of this project during the timestamps above?
Happy to share full Sentry traces or Railway service/project IDs if helpful.
16 Replies
11 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 11 days ago
11 days ago
I'm seeing the same thing. Since Railway automatically applying a CVE patch, my Postgres DB has been unresponsive. I tried redeploying my DB but it's stuck at "creating containers" for >6 minutes. When I go to my DB backups tab, the content isn't loading. Please help, Railway!
trevorblades
I'm seeing the same thing. Since Railway automatically applying a CVE patch, my Postgres DB has been unresponsive. I tried redeploying my DB but it's stuck at "creating containers" for >6 minutes. When I go to my DB backups tab, the content isn't loading. Please help, Railway!
11 days ago
FWIW, my DB is deployed in US West
11 days ago
same here. on US west. my pages are loading slow intermittently. sometimes not at all. this is on prod and staging.
11 days ago
US West currently has issues, and the Team is aware of it. I'd try deploying the service in a different region for now.
11 days ago
I tried deploying my service to US East and it's been migrating the volume for the last 22 minutes. Is this normal? My volume is 3GB in size.
11 days ago
The migration might be affected by the incident as well, I wouldn't recommend making such changes to stateful US-West services during this incident. Don't worry, your data likely isn't lost, Railway makes a backup before performing any migrations, just don't attempt to delete the volume or anything!
0x5b62656e5d
Is the service deployed in US West?
11 days ago
Yeah I solved it by changing the region to southeast asia because most of my users are based there so far it works
dev
The migration might be affected by the incident as well, I wouldn't recommend making such changes to stateful US-West services during this incident. Don't worry, your data likely isn't lost, Railway makes a backup before performing any migrations, just don't attempt to delete the volume or anything!
11 days ago
Appreciate the heads up! Actually the migration seems to have helped a bit. I was getting connection timeout errors specifically on US West, right around when this incident started. Since moving to Southeast Asia it's been a lot quieter.
Also most of my users are in the Philippines anyway so SEA was probably the better region for me long term regardless of the incident.
Data came through fine, nothing lost. Thanks for confirming the backup thing though, good to know.
dev
The migration might be affected by the incident as well, I wouldn't recommend making such changes to stateful US-West services during this incident. Don't worry, your data likely isn't lost, Railway makes a backup before performing any migrations, just don't attempt to delete the volume or anything!
11 days ago
I'm getting mixed messages here. 0x5b62656e5d just said "I'd try deploying the service in a different region" and you're saying "I wouldn't recommend making such changes." That's super confusing...
Good to know that backups have been made. I'll sit tight and wait for a resolution.
trevorblades
I'm getting mixed messages here. 0x5b62656e5d just said "I'd try deploying the service in a different region" and you're saying "I wouldn't recommend making such changes." That's super confusing... Good to know that backups have been made. I'll sit tight and wait for a resolution.
11 days ago
I believe 0x5b62656e5d was referring to services in general, I'm specifically talking about stateful services (those with volumes attached). That's just my opinion though, you're welcome to migrate them over, but I'm seeing a few reports that migrations are getting stuck which is worth considering before migrating
Appreciate the heads up! Actually the migration seems to have helped a bit. I was getting connection timeout errors specifically on US West, right around when this incident started. Since moving to Southeast Asia it's been a lot quieter. Also most of my users are in the Philippines anyway so SEA was probably the better region for me long term regardless of the incident. Data came through fine, nothing lost. Thanks for confirming the backup thing though, good to know.
11 days ago
Glad to hear migrating helped!
dev
I believe 0x5b62656e5d was referring to services in general, I'm specifically talking about stateful services (those with volumes attached). That's just my opinion though, you're welcome to migrate them over, but I'm seeing a few reports that migrations are getting stuck which is worth considering before migrating
11 days ago
Dang, that sucks. My migration definitely seems like it's stuck. Any way to roll it back or stop the process and restore the backup?
trevorblades
Dang, that sucks. My migration definitely seems like it's stuck. Any way to roll it back or stop the process and restore the backup?
11 days ago
Unfortunately no way to do that from the UI that I know of, if it's still stuck once the incident is over then you can make another help thread so a team member may assist you
Status changed to Awaiting User Response Railway • 11 days ago
fvr1
I also did a the DB migration to US East, and it's stuck… what should I do?
11 days ago
If it's still stuck by the time the incident is over then I'd recommend opening another help thread
Status changed to Solved Anonymous • 11 days ago