a month ago
Project ID: 13c7daa2-8572-4ccc-999b-a0fb7b639ae0
Backend service: aa8b15d6-bcc0-4d49-bcb0-7e1929c9eca5
Postgres service: fffe7427-90e9-40a1-a8a0-2ba11c1d3c4d
Environment: production
Issue: All endpoints that query the database hang and eventually return
504 Gateway Timeout (confirmed via browser DevTools, remote address
69.46.46.24:443). The /health endpoint (no DB query) responds instantly
with 200 OK.
Already tried:
- Full redeploy of backend service (new container) - issue persists
- Restarted Postgres service - issue persists
- Verified backend deploy logs are clean (no errors/tracebacks)
- Verified Postgres deploy logs are clean (no FATAL/OOM/too many connections errors)
- Verified Postgres CPU/memory usage is near-zero via Metrics tab
- Ruled out browser cache (incognito mode, same issue)
This looks like an internal networking issue between the backend service
and Postgres on private networking. Requesting help investigating.
7 Replies
a month ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 1 month ago
a month ago
Additional detail: When selecting the affected service in this support
form, the Postgres service showed a "FAILED" status tag, even though its
deploy logs look clean and it's actively accepting connections/completing
checkpoints. This might be a useful clue for the root cause.
a month ago
When you said "logs are clean" do you mean there's no single logs message from deploy logs?
mayori
When you said "logs are clean" do you mean there's no single logs message from deploy logs?
a month ago
Update: I added print statements as the literal first line of both the
login() endpoint function and the get_db() dependency (which runs before
any DB query, on every DB-touching request). Redeployed. Attempted login
again — NEITHER print statement appears in Deploy Logs, even though the
exact same running container logs GET /health and GET /favicon.ico
requests instantly (visible timestamps: 17:48:56).
This confirms the POST /api/auth/login request is not reaching our FastAPI
application code at all — it's being dropped/failing before it's routed to
the container's request handler. This rules out any DB connection issue on
our end; the problem is specifically with how this POST request is (not)
routed to the backend container, despite the same container serving GET
requests normally.
a month ago
Does that POST called from frontend? If so, try make a direct request to the FastAPI using postman or curl, and see if there are Http Logs for that request, if it shows that there are POST request to /api/auth/login, then probably something missed in your frontend code, you may forgot add something like { method: "POST" } when calling the API
mayori
Does that POST called from frontend? If so, try make a direct request to the FastAPI using postman or curl, and see if there are Http Logs for that request, if it shows that there are `POST` request to `/api/auth/login`, then probably something missed in your frontend code, you may forgot add something like `{ method: "POST" }` when calling the API
a month ago
Thanks for the curl suggestion — it helped narrow this down significantly!
Update: Ruled out Chrome specifically — Safari shows the exact same hang.
Both browsers fail identically while curl succeeds in every tested
combination. The key difference is that browsers reuse persistent/keep-alive
connections while curl always opens a fresh connection per request. This
strongly suggests a stale/broken keep-alive connection between your edge
proxy and our backend container (likely left over from earlier redeploys)
that isn't being properly detected and recycled. This looks like a platform-
level connection pooling issue on Railway's gateway, not something we can
fix from the application side.
a month ago
Update: found the actual failure point via nginx logs on our frontend service.
Our frontend is served via nginx, which proxies /api/* requests to the backend using an env var BACKEND_URL set to the backend's public domain (https://backend-production-df78.up.railway.app) — this was deliberately configured to avoid a previously-seen issue with backend.railway.internal caching a stale address across backend restarts.
However, nginx's deploy logs show it is NOT actually connecting to that public domain over the edge. It's resolving to internal private-network addresses on port 8000 (the backend container's internal port), and those connections are timing out:
2026/07/14 20:06:35 [error] 75#75: *68 upstream timed out (110: Operation timed out) while connecting to upstream, client: 100.64.0.11, server: , request: "POST /api/auth/login HTTP/1.1", upstream: "http://10.240.46.209:8000/api/auth/login", host: "bayalag-emeelt-production.up.railway.app", referrer: "https://bayalag-emeelt-production.up.railway.app/login"
2026/07/14 20:07:35 [warn] 75#75: *68 upstream server temporarily disabled while connecting to upstream, client: 100.64.0.11, server: , request: "POST /api/auth/login HTTP/1.1", upstream: "http://[fd12:8969:a5d4:1:a000:43:f9f0:2ed1]:8000/api/auth/login", host: "bayalag-emeelt-production.up.railway.app", referrer: "https://bayalag-emeelt-production.up.railway.app/login"
2026/07/14 20:07:35 [error] 75#75: *68 upstream timed out (110: Operation timed out) while connecting to upstream, client: 100.64.0.11, server: , request: "POST /api/auth/login HTTP/1.1", upstream: "http://[fd12:8969:a5d4:1:a000:43:f9f0:2ed1]:8000/api/auth/login", host: "bayalag-emeelt-production.up.railway.app", referrer: "https://bayalag-emeelt-production.up.railway.app/login"
Two things stand out:
Even though proxy_pass is configured with the public https:// domain, the actual upstream connection is going out over what appears to be Railway's private network (10.240.x.x / fd12:... ULA range), not the public edge on 443.
The resolved address changes between attempts (IPv4 → IPv6) and each one times out — so this isn't a one-time stale cache, the private-network path itself seems to be intermittently unreachable right now.
This matches our earlier finding that curl from an external machine hits the same backend instantly and correctly — because that traffic never touches this private-network path at all.
Given this looks like server-to-server (service-to-service, same project) traffic to a public Railway domain is being silently routed over private networking, and that private-network hop is timing out, this really does look like it's on Railway's side rather than something we can fix in our app code. Would appreciate someone from the team taking a look at the private networking path between these two services.
a month ago
Resolved. Root cause confirmed and fixed on our end — it wasn't a backend/Postgres issue after all.
Our frontend runs nginx as a reverse proxy for /api/* requests to the backend. Turns out nginx caches DNS resolution, and Railway's private network IP for the backend changes on every redeploy — so nginx kept trying to connect to a stale/dead address and timing out (504), while the backend itself was completely healthy the whole time (which is why direct curl requests always worked fine).
Found two older threads on this exact issue confirming it's a known nginx + Railway private networking gotcha:
https://station.railway.com/questions/nginx-with-private-networking-upstream-8d7ce3c3
https://station.railway.com/questions/reliability-internal-networking-7daff8bd
Switched our frontend's reverse proxy from nginx to Caddy (does a fresh DNS lookup on every request, no caching) and the issue is gone. Thanks for the help pointing us toward isolating this with curl earlier, @mayori — that was what made it clear the backend itself was never the problem.
