Backend cannot reach Postgres - 504 Gateway Timeout on all DB-touching endpoints"
mmudan79-ops
HOBBYOP

a month ago

Project ID: 13c7daa2-8572-4ccc-999b-a0fb7b639ae0

Backend service: aa8b15d6-bcc0-4d49-bcb0-7e1929c9eca5

Postgres service: fffe7427-90e9-40a1-a8a0-2ba11c1d3c4d

Environment: production

Issue: All endpoints that query the database hang and eventually return

504 Gateway Timeout (confirmed via browser DevTools, remote address

69.46.46.24:443). The /health endpoint (no DB query) responds instantly

with 200 OK.

Already tried:

  • Full redeploy of backend service (new container) - issue persists
  • Restarted Postgres service - issue persists
  • Verified backend deploy logs are clean (no errors/tracebacks)
  • Verified Postgres deploy logs are clean (no FATAL/OOM/too many connections errors)
  • Verified Postgres CPU/memory usage is near-zero via Metrics tab
  • Ruled out browser cache (incognito mode, same issue)

This looks like an internal networking issue between the backend service

and Postgres on private networking. Requesting help investigating.

$10 Bounty

7 Replies

Railway
BOT

a month ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway about 1 month ago


mmudan79-ops
HOBBYOP

a month ago

Additional detail: When selecting the affected service in this support

form, the Postgres service showed a "FAILED" status tag, even though its

deploy logs look clean and it's actively accepting connections/completing

checkpoints. This might be a useful clue for the root cause.


a month ago

When you said "logs are clean" do you mean there's no single logs message from deploy logs?


mayori

When you said "logs are clean" do you mean there's no single logs message from deploy logs?

mmudan79-ops
HOBBYOP

a month ago

Update: I added print statements as the literal first line of both the

login() endpoint function and the get_db() dependency (which runs before

any DB query, on every DB-touching request). Redeployed. Attempted login

again — NEITHER print statement appears in Deploy Logs, even though the

exact same running container logs GET /health and GET /favicon.ico

requests instantly (visible timestamps: 17:48:56).

This confirms the POST /api/auth/login request is not reaching our FastAPI

application code at all — it's being dropped/failing before it's routed to

the container's request handler. This rules out any DB connection issue on

our end; the problem is specifically with how this POST request is (not)

routed to the backend container, despite the same container serving GET

requests normally.


a month ago

Does that POST called from frontend? If so, try make a direct request to the FastAPI using postman or curl, and see if there are Http Logs for that request, if it shows that there are POST request to /api/auth/login, then probably something missed in your frontend code, you may forgot add something like { method: "POST" } when calling the API


mayori

Does that POST called from frontend? If so, try make a direct request to the FastAPI using postman or curl, and see if there are Http Logs for that request, if it shows that there are `POST` request to `/api/auth/login`, then probably something missed in your frontend code, you may forgot add something like `{ method: "POST" }` when calling the API

mmudan79-ops
HOBBYOP

a month ago

Thanks for the curl suggestion — it helped narrow this down significantly!

Update: Ruled out Chrome specifically — Safari shows the exact same hang.

Both browsers fail identically while curl succeeds in every tested

combination. The key difference is that browsers reuse persistent/keep-alive

connections while curl always opens a fresh connection per request. This

strongly suggests a stale/broken keep-alive connection between your edge

proxy and our backend container (likely left over from earlier redeploys)

that isn't being properly detected and recycled. This looks like a platform-

level connection pooling issue on Railway's gateway, not something we can

fix from the application side.


mmudan79-ops
HOBBYOP

a month ago

Update: found the actual failure point via nginx logs on our frontend service.

Our frontend is served via nginx, which proxies /api/* requests to the backend using an env var BACKEND_URL set to the backend's public domain (https://backend-production-df78.up.railway.app) — this was deliberately configured to avoid a previously-seen issue with backend.railway.internal caching a stale address across backend restarts.

However, nginx's deploy logs show it is NOT actually connecting to that public domain over the edge. It's resolving to internal private-network addresses on port 8000 (the backend container's internal port), and those connections are timing out:

2026/07/14 20:06:35 [error] 75#75: *68 upstream timed out (110: Operation timed out) while connecting to upstream, client: 100.64.0.11, server: , request: "POST /api/auth/login HTTP/1.1", upstream: "http://10.240.46.209:8000/api/auth/login", host: "bayalag-emeelt-production.up.railway.app", referrer: "https://bayalag-emeelt-production.up.railway.app/login"

2026/07/14 20:07:35 [warn] 75#75: *68 upstream server temporarily disabled while connecting to upstream, client: 100.64.0.11, server: , request: "POST /api/auth/login HTTP/1.1", upstream: "http://[fd12:8969:a5d4:1:a000:43:f9f0:2ed1]:8000/api/auth/login", host: "bayalag-emeelt-production.up.railway.app", referrer: "https://bayalag-emeelt-production.up.railway.app/login"

2026/07/14 20:07:35 [error] 75#75: *68 upstream timed out (110: Operation timed out) while connecting to upstream, client: 100.64.0.11, server: , request: "POST /api/auth/login HTTP/1.1", upstream: "http://[fd12:8969:a5d4:1:a000:43:f9f0:2ed1]:8000/api/auth/login", host: "bayalag-emeelt-production.up.railway.app", referrer: "https://bayalag-emeelt-production.up.railway.app/login"

Two things stand out:

Even though proxy_pass is configured with the public https:// domain, the actual upstream connection is going out over what appears to be Railway's private network (10.240.x.x / fd12:... ULA range), not the public edge on 443.

The resolved address changes between attempts (IPv4 → IPv6) and each one times out — so this isn't a one-time stale cache, the private-network path itself seems to be intermittently unreachable right now.

This matches our earlier finding that curl from an external machine hits the same backend instantly and correctly — because that traffic never touches this private-network path at all.

Given this looks like server-to-server (service-to-service, same project) traffic to a public Railway domain is being silently routed over private networking, and that private-network hop is timing out, this really does look like it's on Railway's side rather than something we can fix in our app code. Would appreciate someone from the team taking a look at the private networking path between these two services.


mmudan79-ops
HOBBYOP

a month ago

Resolved. Root cause confirmed and fixed on our end — it wasn't a backend/Postgres issue after all.

Our frontend runs nginx as a reverse proxy for /api/* requests to the backend. Turns out nginx caches DNS resolution, and Railway's private network IP for the backend changes on every redeploy — so nginx kept trying to connect to a stale/dead address and timing out (504), while the backend itself was completely healthy the whole time (which is why direct curl requests always worked fine).

Found two older threads on this exact issue confirming it's a known nginx + Railway private networking gotcha:

https://station.railway.com/questions/nginx-with-private-networking-upstream-8d7ce3c3

https://station.railway.com/questions/reliability-internal-networking-7daff8bd

Switched our frontend's reverse proxy from nginx to Caddy (does a fresh DNS lookup on every request, no caching) and the issue is gone. Thanks for the help pointing us toward isolating this with curl earlier, @mayori — that was what made it clear the backend itself was never the problem.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...