a month ago
Since ~07:30 UTC today (Aug 24), all non-HTTPS paths to our databases stopped working. HTTPS paths (dashboard, deployed apps, Console) work fine throughout.
1. TCP proxy — timed out on both environments
- Production: nozomi.proxy.rlwy.net:34660 → connection timed out
- Staging: caboose.proxy.rlwy.net:18053 → connection timed out
- Port 443 to the same hostnames connects fine (verified with nc)
- Confirmed from two independent networks: residential ISP (Jakarta, Indonesia) and a separate cloud sandbox environment — so this is not a local firewall/ISP issue
- Both proxies had been working normally earlier the same day
2. SSH tunnel — timed out
$ railway connect Postgres --tunnel-only
Using SSH key from file /Users/.../.ssh/id_ed25519.pub
Timed out waiting for the SSH tunnel to become ready on 127.0.0.1
- CLI v5.43.1 (latest, confirmed before retrying)
- Correct project/environment/service linked (production / Postgres)
Timeline note
A backup named "Pre-Security-Patch Backup" appeared on our production Postgres service ~Aug 23 (1 day before the breakage) that nobody on our team created. If that was a platform-side security patch, the timing coincides exactly with this connectivity breakage — possibly relevant for root cause.
What we've ruled out
- Local network (two independent networks tested)
- CLI version (5.43.1)
- Proxy configuration (dashboard showed it as normal throughout)
- Railway status page (fully operational, no incidents listed)
Current state
- We deleted the production TCP proxy intentionally after the breakage (keeping the DB private-only) — the issue predates the deletion. The staging proxy still exists and still times out.
- Working around via the web Console + file upload, but our normal workflow (external clients, agent tooling, scheduled dumps via tunnel) is fully blocked.
Possibly related: "Unable to connect to the [Postgres] database via SSH: WebSocket tunnel failed to connect" (open bounty, ~1 month old) — similar tunnel failure, but in our case confirmed from two independent networks with no local firewall restrictions, so it does not appear to be network-side.
Project: inspiring-cooperation (cdb403cc-8e5f-4dfe-b6b8-d59741b71d6c) · Region: Southeast Asia
Postgres 18.4 · What we need: railway connect tunnel working again.
1 Replies
a month ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 1 month ago
17 days ago
Solid writeup, and testing from two networks rules out the usual suspects. A few things worth checking before this gets pinned on the platform.
Is public networking still enabled on the Postgres services themselves? Databases are private by default, and a proxy timing out at TCP level while the dashboard shows everything normal smells like the service lost its public network attachment even though the proxy entry still renders.
Also worth posting the exact output of nc -vz against the proxy host on its real port versus port 443, since 443 connecting while an arbitrary port times out on the same hostname does look like platform side forwarding.
One more thing: that Pre-Security-Patch Backup nobody on your team created deserves more than a timeline note. If the name came from the platform, some maintenance probably ran on your region's storage, but if it didn't, treat that as an incident and report it to Railway directly with your project id.
Either way, worth asking them straight up whether any planned maintenance or snapshot ran on SEA Postgres volumes around Aug 23. And since this is clearly platform side, try their Discord too, staff tend to respond way faster there than on the forum for live outage type stuff.