502 "connection refused" on 100% of public requests — edge can't reach healthy container (private network works fine)
jtinajero21
HOBBYOP

a month ago

Summary: our api service's public domain has been returning 502 "connection refused" on every single request, for every path, for 3+ days. The container itself is healthy — requests over the private network (from another service in the same project) succeed normally. This looks like a stale/broken routing entry for the public domain, not an application issue.

Project: MetaAds-AI

Service: api (a88b6b8a-2ca0-4048-8c79-30053d341c27)

Environment: production

Affected domain: api-production-c5df.up.railway.app

Current deployment: fb919bf0-6d38-4313-936e-14ca6fc4eefb (status: SUCCESS, healthy)

What's happening:

Every public request to the domain above fails with 502, regardless of path:

  • GET /meta/oauth/callback -> 502 "connection refused"
  • GET /favicon.ico -> 502 "connection refused"
  • GET /robots.txt -> 502 "connection refused"

This has been reproduced across multiple separate days (Jul 23, 24, 25) and different times of day, always with the same result. Both replica instances fail simultaneously (ruled out a single bad instance).

What I've already checked/tried:

  1. App is listening correctly: Dockerfile CMD runs uvicorn app.main:app --host 0.0.0.0 --port 8000, confirmed in deploy logs ("Uvicorn running on http://0.0.0.0:8000"). No port/host mismatch — Target Port for the domain also confirmed at default 8000, matching.

  2. Ruled out DNS: checked the new Network Logs -> DNS panel for this deployment. Zero failed queries. Only internal lookups (timescaledb.railway.internal) which all resolve correctly.

  3. Ruled out app-level bug: requests reaching this same service over the private network (RAILWAY_PRIVATE_DOMAIN, from our web service) succeed normally — /clinics, /organizations, /auth/login, /meta/status all return 200 OK. The container is healthy and serving traffic; only the public domain's edge path is broken.

  4. Ruled out capacity/resource exhaustion: CPU/memory were near-idle at failure time, scaled to 2 replicas, both fail identically.

  5. Debug header check (X-Railway-Debug: 1) on the failing requests shows:

    x-railway-fallback: true

    x-railway-upstream-zone: railway/us-west2

    x-railway-edge: mia1

    This indicates the edge (Miami) is falling back because it can't reach the upstream in us-west2.

  6. Tried moving the service's region temporarily (sfo -> us-east1/iad -> back to sfo, per a suggestion from a previous thread) and redeploying both times (both deployments reached SUCCESS). This did not change anything — same 502, same fallback signature, same upstream-zone mismatch, even while the service was actively running in a different region.

Given the domain fails 100% of the time regardless of path, across days, and survives a region migration unchanged, this points to the public domain itself having a stale internal routing/service-discovery association that isn't being refreshed by normal redeploys or region changes.

Question: could someone check whether api-production-c5df.up.railway.app has a broken/stale route entry, and rebuild it on your end? If regenerating the domain from our side is the right move, happy to do that — just wanted to flag it here first in case there's a known issue or a faster fix on your side.

$10 Bounty

3 Replies

Railway
BOT

a month ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway • 26 days ago

jtinajero21
HOBBYOP

a month ago

Update: I tried to work around this myself by removing the domain and generating a fresh one, and found something that I think is the actual root cause.

I set serviceDomains.api-production-c5df.up.railway.app to null via the config API and committed the change. The API responded with "Service has been updated and changes are staged for deployment" and the deployment triggered successfully. But when I re-fetched the service config afterward, the domain was still there, unchanged — the removal never actually took effect.

This isn't a one-off: we hit the exact same behavior a few days ago with a different (duplicate) domain on this same service, api-production-bc56.up.railway.app — every attempt to remove it also reported success but the domain kept reappearing in the config.

So on this specific service, networking config changes (at least domain removal) report success via the API but don't persist. That would also explain why the region migration (sfo -> us-east1 -> sfo) didn't fix the 502/fallback issue: if this service's networking state is stuck, changing the region wouldn't touch whatever's actually broken.

Could someone check this service's networking/routing state directly on the backend and rebuild it manually? Self-service domain removal/regeneration isn't working for us here.

Service: api (a88b6b8a-2ca0-4048-8c79-30053d341c27)

Project: MetaAds-AI

Environment: production

Domain stuck / not removable: api-production-c5df.up.railway.app

Previously stuck / not removable: api-production-bc56.up.railway.app

jtinajero21
HOBBYOP

a month ago

Update: I created a brand-new service from scratch to rule out anything specific to our original "api" service, and got the same failure.

New service: api-v2 (82f8d5a5-543f-4ab0-b966-d6a7922ea689), same repo/branch, freshly created today, deployed only in us-east1 (iad), 1 replica, brand-new domain (api-v2-production-32ea.up.railway.app), deployment fully SUCCESS.

curl -sD - -o /dev/null -H "X-Railway-Debug: 1" "https://api-v2-production-32ea.up.railway.app/health"

Response:

HTTP/2 502

content-type: application/json

server: railway-hikari

x-railway-fallback: true

x-railway-request-id: 12YX8o88SjW-SkDVO8poTA

x-railway-upstream-zone: railway/us-east4-eqdc4a

x-railway-edge: mia1

x-hikari-trace: mia1.0twb

Same exact signature as the original "api" service: fallback true, edge mia1 unable to reach the upstream zone (this time us-east4-eqdc4a, since this service is in iad). This rules out anything project/service/domain-specific — a completely fresh service, fresh domain, different region, still fails identically from the same edge.

I also checked status.railway.com: for July, "Networking — Public" shows 99.42% uptime for both US East and US West, while every other category (Builds, Deployments, Compute, Storage, Private Networking) shows 100%. That matches exactly what we're seeing — only public networking is degraded, and only in the US regions, which is where mia1 would be routing.

At this point I don't think there's anything else I can try from my side — new service, new domain, new region, same result. Could someone look into whether the mia1 edge (or its routing to US East/West upstreams) has a known ongoing issue? This has now been reproducible for 3+ days across two completely separate services.

Project: MetaAds-AI

Affected/tested services: api (a88b6b8a-2ca0-4048-8c79-30053d341c27, sfo) and api-v2 (82f8d5a5-543f-4ab0-b966-d6a7922ea689, iad)

Environment: production

msns-dev
FREE

a month ago

Your application architecture must be upgraded to leverage Gunicorn as a process manager with Uvicorn workers.

  1. Replace the standalone uvicorn command in your Dockerfile with the following configuration:

      RUN pip install gunicorn uvicorn

  2. Bind to 0.0.0.0 using Gunicorn with the Uvicorn worker class. Set keep-alive well above standard proxy timeouts.

    CMD ["gunicorn", "app.main:app", "-k", "uvicorn.workers.UvicornWorker", "--bind", "0.0.0.0:8000", "--workers", "2", "--keep-alive", "75"]

Welcome!

Sign in to your Railway account to join the conversation.

Login
Loading...