Intermittent 502 "connection refused" on /meta/oauth/callback — edge fallback detected (mia1 → us-west2)
jtinajero21
HOBBYOP

a month ago

Project: MetaAds-AI (53ff9c2a-cec4-4a7b-bb2d-38a593f785c8)

Service: api (a88b6b8a-2ca0-4048-8c79-30053d341c27)

Public domain: api-production-c5df.up.railway.app

Affected deployment: f5da864b-f2bd-469e-be5d-b5a2e9dd1249 (commit e619eab, sfo region, 2 replicas)

Problem:

GET /meta/oauth/callback (our public Meta OAuth callback route) consistently returns 502, while every other route on the same service and domain (/clinics, /auth/me, /meta/oauth/login, etc.) responds 200 normally at the same time.

Evidence collected:

Railway's HTTP logs show "error":"connection refused" against the active replica(s), with 0-2ms duration, repeated across multiple separate deployments (even after scaling from 1 to 2 replicas and redeploying from scratch).

Deploy logs show zero entries for this route — not even an error log — indicating the request never reaches the application process.

Repeating the request with curl -H "X-Railway-Debug: 1" returns: HTTP/2 502, x-railway-fallback: true, x-railway-upstream-zone: railway/us-west2, x-railway-edge: mia1, x-hikari-trace: mia1.nv6z, x-railway-request-id: On85EQNqSTSxee5d2h0iww. The x-railway-fallback: true header indicates the edge node (mia1, Miami) could not reach the actual upstream (us-west2).

What we've already ruled out:

Not an app bug: reviewed the route code (fully async, no blocking calls) and rewrote our timing middleware from BaseHTTPMiddleware to pure ASGI just in case — the problem persisted identically after that fix and redeploy.

Not a resource issue: CPU/memory metrics at the time of failure show minimal usage (~0.14GB of 8GB, ~0.05 vCPU of 8).

Not an app config issue: META_REDIRECT_URI and FRONTEND_URL are set correctly, no copy-paste errors.

We tried removing a duplicate domain (api-production-bc56.up.railway.app) left over from a previous GitHub repo reconnection — the removal is confirmed as "applied," but the domain keeps reappearing in the service config after committing the change.

Question for support: why can't the edge node in mia1 reach the upstream in us-west2 for this specific service/deployment, and why won't the duplicate domain removal persist?

$10 Bounty

2 Replies

Railway
BOT

a month ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway 29 days ago


jtinajero21
HOBBYOP

a month ago

Hi, thanks for the suggestion — I tried it but the issue is still happening exactly the same way.

What I did:

  1. Changed the api service's region from sfo to us-east1 (iad), committed, and redeployed. Deployment reached SUCCESS.
  2. Changed it back to sfo, committed, and redeployed again. Deployment also reached SUCCESS (deployment id: fb919bf0-6d38-4313-936e-14ca6fc4eefb).

After that, I retried the request that was failing (GET /meta/oauth/callback on api-production-c5df.up.railway.app) and it's still returning 502 "connection refused" on both replica instances at the same time. The app's own deploy logs show no trace of the request ever arriving — it's failing before it reaches the container.

I ran the debug header check again just now:

curl -sD - -o /dev/null -H "X-Railway-Debug: 1" "https://api-production-c5df.up.railway.app/meta/oauth/callback?code=x&state=y"

Response:

HTTP/2 502

content-type: application/json

server: railway-hikari

x-railway-fallback: true

x-railway-request-id: 0z6b2Wz0Qpmrp9_UwUFZXw

x-railway-upstream-zone: railway/us-west2

x-railway-edge: mia1

x-hikari-trace: mia1.nv6z

Same signature as before the region change: x-railway-fallback: true, edge mia1 (Miami) unable to reach upstream zone railway/us-west2. The region toggle didn't change this at all.

Project: MetaAds-AI

Service: api (a88b6b8a-2ca0-4048-8c79-30053d341c27)

Environment: production

Is there another edge/routing reset you can trigger on your side, or is this a broader mia1 -> us-west2 routing problem?


Status changed to Open 0x5b62656e5d 26 days ago


ayalaa12
FREETop 5% Contributor

13 days ago

One part of the interpretation is worth correcting first: X-Railway-Edge: mia1 and X-Railway-Upstream-Zone: railway/us-west2 are not themselves a region mismatch. Railway's current edge documentation says the first is the ingress POP and the second is the deployment region; entering at Miami and being served from US West is expected for an SFO-only deployment.

What is abnormal is the paired evidence: the same host and active deployment serve other paths, while this path returns a 0–2 ms Railway 502 with responseDetails=connection refused and no application log. That excludes a general PORT/listen failure and makes a paired control test more useful than another region toggle.

For one test window, capture two requests from the same client within a few seconds:

curl -sS -D callback.headers -o /dev/null \
  -H 'X-Railway-Debug: 1' \
  'https://api-production-c5df.up.railway.app/meta/oauth/callback?code=diagnostic&state=diagnostic'

curl -sS -D control.headers -o /dev/null \
  -H 'X-Railway-Debug: 1' \
  'https://api-production-c5df.up.railway.app/meta/oauth/login'

Use only fake diagnostic values, never a real OAuth code/state. In Railway HTTP logs, filter each X-Railway-Request-Id and record host, path, httpStatus, responseDetails, upstreamErrors, edgeRegion, deploymentId/replica and upstreamRqDuration. Railway documents request-ID, path, response-details and edge-region filters here: https://docs.railway.com/observability/logs

If the callback alone still fails, add a temporary alias path in the application (for example /__diag/meta-oauth-callback) that invokes the same handler and repeat once. The outcomes are decisive:

  • alias reaches the app but the original path returns an edge 502: path-specific edge/routing state;
  • both paths fail on the same deployment/replica while the control succeeds: route/target metadata for that handler path needs Railway tracing;
  • both paths reach the app: the earlier failure was transient and the remaining issue is inside the handler/outbound OAuth flow.

The duplicate Railway domain should also be deleted as a domain object, not updated to null or merely removed from staged config:

railway domain list --service api --environment production --project 53ff9c2a-cec4-4a7b-bb2d-38a593f785c8 --json
railway domain status api-production-bc56.up.railway.app --service api --environment production --project 53ff9c2a-cec4-4a7b-bb2d-38a593f785c8
railway domain delete api-production-bc56.up.railway.app --service api --environment production --project 53ff9c2a-cec4-4a7b-bb2d-38a593f785c8
railway domain list --service api --environment production --project 53ff9c2a-cec4-4a7b-bb2d-38a593f785c8 --json

Omit --yes so the CLI confirms the exact object. Current CLI documentation says domain delete removes a custom or service domain by name/URL/ID: https://docs.railway.com/cli/domain

If the final list confirms deletion but the object later reappears, or the paired request IDs show a callback-only edge 502, send Railway the two request IDs, exact UTC timestamp, deployment/replica IDs, before/after domain list, and the alias result. That is a minimal reproducible control-plane/edge trace; another region migration would add noise without testing the path-specific behavior.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...