a month ago
Project: MetaAds-AI (53ff9c2a-cec4-4a7b-bb2d-38a593f785c8)
Service: api (a88b6b8a-2ca0-4048-8c79-30053d341c27)
Public domain: api-production-c5df.up.railway.app
Affected deployment: f5da864b-f2bd-469e-be5d-b5a2e9dd1249 (commit e619eab, sfo region, 2 replicas)
Problem:
GET /meta/oauth/callback (our public Meta OAuth callback route) consistently returns 502, while every other route on the same service and domain (/clinics, /auth/me, /meta/oauth/login, etc.) responds 200 normally at the same time.
Evidence collected:
Railway's HTTP logs show "error":"connection refused" against the active replica(s), with 0-2ms duration, repeated across multiple separate deployments (even after scaling from 1 to 2 replicas and redeploying from scratch).
Deploy logs show zero entries for this route — not even an error log — indicating the request never reaches the application process.
Repeating the request with curl -H "X-Railway-Debug: 1" returns: HTTP/2 502, x-railway-fallback: true, x-railway-upstream-zone: railway/us-west2, x-railway-edge: mia1, x-hikari-trace: mia1.nv6z, x-railway-request-id: On85EQNqSTSxee5d2h0iww. The x-railway-fallback: true header indicates the edge node (mia1, Miami) could not reach the actual upstream (us-west2).
What we've already ruled out:
Not an app bug: reviewed the route code (fully async, no blocking calls) and rewrote our timing middleware from BaseHTTPMiddleware to pure ASGI just in case — the problem persisted identically after that fix and redeploy.
Not a resource issue: CPU/memory metrics at the time of failure show minimal usage (~0.14GB of 8GB, ~0.05 vCPU of 8).
Not an app config issue: META_REDIRECT_URI and FRONTEND_URL are set correctly, no copy-paste errors.
We tried removing a duplicate domain (api-production-bc56.up.railway.app) left over from a previous GitHub repo reconnection — the removal is confirmed as "applied," but the domain keeps reappearing in the service config after committing the change.
Question for support: why can't the edge node in mia1 reach the upstream in us-west2 for this specific service/deployment, and why won't the duplicate domain removal persist?
2 Replies
a month ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 29 days ago
a month ago
Hi, thanks for the suggestion — I tried it but the issue is still happening exactly the same way.
What I did:
- Changed the
apiservice's region from sfo to us-east1 (iad), committed, and redeployed. Deployment reached SUCCESS. - Changed it back to sfo, committed, and redeployed again. Deployment also reached SUCCESS (deployment id: fb919bf0-6d38-4313-936e-14ca6fc4eefb).
After that, I retried the request that was failing (GET /meta/oauth/callback on api-production-c5df.up.railway.app) and it's still returning 502 "connection refused" on both replica instances at the same time. The app's own deploy logs show no trace of the request ever arriving — it's failing before it reaches the container.
I ran the debug header check again just now:
curl -sD - -o /dev/null -H "X-Railway-Debug: 1" "https://api-production-c5df.up.railway.app/meta/oauth/callback?code=x&state=y"
Response:
HTTP/2 502
content-type: application/json
server: railway-hikari
x-railway-fallback: true
x-railway-request-id: 0z6b2Wz0Qpmrp9_UwUFZXw
x-railway-upstream-zone: railway/us-west2
x-railway-edge: mia1
x-hikari-trace: mia1.nv6z
Same signature as before the region change: x-railway-fallback: true, edge mia1 (Miami) unable to reach upstream zone railway/us-west2. The region toggle didn't change this at all.
Project: MetaAds-AI
Service: api (a88b6b8a-2ca0-4048-8c79-30053d341c27)
Environment: production
Is there another edge/routing reset you can trigger on your side, or is this a broader mia1 -> us-west2 routing problem?
Status changed to Open 0x5b62656e5d • 26 days ago
13 days ago
One part of the interpretation is worth correcting first: X-Railway-Edge: mia1 and X-Railway-Upstream-Zone: railway/us-west2 are not themselves a region mismatch. Railway's current edge documentation says the first is the ingress POP and the second is the deployment region; entering at Miami and being served from US West is expected for an SFO-only deployment.
What is abnormal is the paired evidence: the same host and active deployment serve other paths, while this path returns a 0–2 ms Railway 502 with responseDetails=connection refused and no application log. That excludes a general PORT/listen failure and makes a paired control test more useful than another region toggle.
For one test window, capture two requests from the same client within a few seconds:
curl -sS -D callback.headers -o /dev/null \
-H 'X-Railway-Debug: 1' \
'https://api-production-c5df.up.railway.app/meta/oauth/callback?code=diagnostic&state=diagnostic'
curl -sS -D control.headers -o /dev/null \
-H 'X-Railway-Debug: 1' \
'https://api-production-c5df.up.railway.app/meta/oauth/login'Use only fake diagnostic values, never a real OAuth code/state. In Railway HTTP logs, filter each X-Railway-Request-Id and record host, path, httpStatus, responseDetails, upstreamErrors, edgeRegion, deploymentId/replica and upstreamRqDuration. Railway documents request-ID, path, response-details and edge-region filters here: https://docs.railway.com/observability/logs
If the callback alone still fails, add a temporary alias path in the application (for example /__diag/meta-oauth-callback) that invokes the same handler and repeat once. The outcomes are decisive:
- alias reaches the app but the original path returns an edge 502: path-specific edge/routing state;
- both paths fail on the same deployment/replica while the control succeeds: route/target metadata for that handler path needs Railway tracing;
- both paths reach the app: the earlier failure was transient and the remaining issue is inside the handler/outbound OAuth flow.
The duplicate Railway domain should also be deleted as a domain object, not updated to null or merely removed from staged config:
railway domain list --service api --environment production --project 53ff9c2a-cec4-4a7b-bb2d-38a593f785c8 --json
railway domain status api-production-bc56.up.railway.app --service api --environment production --project 53ff9c2a-cec4-4a7b-bb2d-38a593f785c8
railway domain delete api-production-bc56.up.railway.app --service api --environment production --project 53ff9c2a-cec4-4a7b-bb2d-38a593f785c8
railway domain list --service api --environment production --project 53ff9c2a-cec4-4a7b-bb2d-38a593f785c8 --jsonOmit --yes so the CLI confirms the exact object. Current CLI documentation says domain delete removes a custom or service domain by name/URL/ID: https://docs.railway.com/cli/domain
If the final list confirms deletion but the object later reappears, or the paired request IDs show a callback-only edge 502, send Railway the two request IDs, exact UTC timestamp, deployment/replica IDs, before/after domain list, and the alias result. That is a minimal reproducible control-plane/edge trace; another region migration would add noise without testing the path-specific behavior.