4 days ago
Project details
Project: lively-encouragement (ID: 044ac1b4-2864-4404-83c8-a9dfef896ed8)
Environment: production (ID: 87417b6f-034c-429b-bc5f-735c4d74103c)
Service: web (ID: 29a3e6e2-fff8-4138-ad9e-e104bf8aa7ae)
Region: sfo
Custom domain: holfly.com (via Cloudflare)
Railway domain: web-production-fc1f3.up.railway.app
Symptom
Since ~2026-07-03 13:20 UTC: browser gets Cloudflare 524 (origin timeout). Direct curl confirms TCP connects instantly but no HTTP response ever arrives. Identical hang on the raw Railway subdomain, ruling out DNS/Cloudflare/custom-domain as the cause.
Already checked/ruled out
Container health — logs show clean startup and healthy 5-min scheduler cycles throughout, zero errors.
Two redeploys, both produced RUNNING healthy instances, neither fixed external reachability: aeed0b8b-f017-4dc2-a08f-e280c0118b9d (fresh build, 14:02:49Z) and 30bbfd47-5fd2-496c-ac16-5fb3068b96c0 (manual restart, 14:40:05Z).
Database — checked via public proxy: zero locks, zero long-running transactions, only our own diagnostic connection.
Note: our app connects to Postgres via the private internal network (postgres-u-ei.railway.internal), a different path than the public-proxy check above, so we couldn't verify that specific path ourselves.
Why we suspect private networking
Your status page reported an incident earlier today (~10:00–10:30 UTC) citing "elevated latency and packet loss on Edge Network" and "private networking between regions" — this outage reproduces the same signature (origin healthy internally, external requests time out ~100s), and per your own status page, isolated recurrences may not show up on the public dashboard.
Requesting
Check private networking/internal routing health for service 29a3e6e2-fff8-4138-ad9e-e104bf8aa7ae in sfo.
Check whether requests are reaching the container's port or being dropped at the edge/proxy layer.
ETA if this is a known issue.
1 Replies
Status changed to Awaiting Railway Response Railway • 4 days ago
4 days ago
Update: isolated the fault to the production environment itself, not a specific service/deployment
Since posting this, we ran a decisive test: created a brand new, unrelated service in the same project (web-v2, service ID ce2325d7-ba11-4214-862d-d90e2e12a7f5) in the same production environment, deployed fresh from the same repo, pointed at the same database, and generated a fresh Railway-provided domain for it (web-v2-production-3e19.up.railway.app).
Result: identical failure. Its own logs show a completely clean boot — gunicorn started, Database initialised succeeded (confirming it can reach Postgres over the private network fine), scheduler loaded all 3 profiles from the DB successfully. But external HTTP requests to its dedicated domain hang exactly the same way as the original web service — TCP connects instantly, then no response, timing out.
For comparison, our staging environment in the same project/account continues to work normally (HTTP 200, ~0.5s) throughout this entire incident.
This rules out:
The specific container/instance (this is a brand new one)
The specific deployment/build (freshly built moments ago)
Application code (identical clean boot, DB connection succeeds)
Database connectivity (confirmed working from inside the container)
Account-wide outage (staging in the same account is unaffected)
This isolates the problem specifically to the production environment's edge/routing layer (environment ID 87417b6f-034c-429b-bc5f-735c4d74103c) — every service placed into this environment becomes externally unreachable, regardless of which service, build, or deployment it is. Something in how this specific environment routes external traffic to its containers appears to be broken at the platform level.
Given this is now affecting two independent services identically, could someone take a look at the routing/proxy configuration specifically for environment 87417b6f-034c-429b-bc5f-735c4d74103c in the sfo region? Happy to provide the new service ID or any further logs.
3 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 3 days ago