3 months ago
We're experiencing an issue where if a request is forwarded by a reverse proxy (kong, hosted in a railway project) to an upstream service (hosted in another railway project), the response we're getting is:
{
"status": "error",
"code": 404,
"message": "Application not found",
"request_id": "Ry17M9MpQPWAG3Ck0ubPiw"
}We tried to directly access the custom domain and, for good measure, a generated domain as well. The behavior is as follows:
Request -> Service (either via custom domain or generated) ... ✅
Request -> Kong (via custom domain) -> Service (either via custom domain or generated) ... ❌ (application not found)
This was known to be last working today around 12AM PST
Attachments
21 Replies
3 months ago
Please have kong strip the fastly-ff header.
this is also happening to us,
we have a nextjs proxy to an /api/
both are in railway, but having failure
Attachments
3 months ago
Please strip the fastly-ff header before making a request to the upstream.
3 months ago
It's an incoming header, delete it.
Question, our clients are asking why suddenly this need to fix this?
do we have an incident report for this?
or is this part of the edge network incident this morning?
3 months ago
It's a new header that Fastly is sending.
3 months ago
Sameissuehere.Customdomainsreturning404"Applicationnotfound" since~6:10PMPST.Direct*.up.railway.app URLsworkfine.Alreadytriedredeployingandre-addingcustomdomainwithfreshCNAME+TXTverification(bothgreen).Region:us-west2.
https://station.railway.com/questions/custom-domains-returning-504-timeout-e-9b9ba1b2
{
"method": "GET",
"path": "/",
"headers": {
"host": "web1-dev.up.railway.app",
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36",
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
"accept-encoding": "gzip",
"accept-language": "en-US,en;q=0.9",
"backend_is_origin": "1",
"cache-control": "no-cache",
"cdn-loop": "Fastly",
"fastly-client-ip": "180.190.225.237",
"fastly-ff": "scJ4iSV4UkyHDI2juJNzrcUzYk/oO1PMCyYg4bz5Utg=!TYO!cache-tyo11947-TYO",
"fastly-orig-accept-encoding": "gzip, deflate, br, zstd",
"fastly-ssl": "1",
"pragma": "no-cache",
"priority": "u=0, i",
"sec-ch-ua": "\"Chromium\";v=\"145\", \"Not:A-Brand\";v=\"99\"",
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": "\"macOS\"",
"sec-fetch-dest": "document",
"sec-fetch-mode": "navigate",
"sec-fetch-site": "none",
"sec-fetch-user": "?1",
"upgrade-insecure-requests": "1",
"x-edge-region": "Asia",
"x-forwarded-for": "180.190.225.237, 167.82.139.47",
"x-forwarded-host": "web1-dev.up.railway.app",
"x-forwarded-proto": "https",
"x-forwarded-server": "cache-tyo11947-TYO",
"x-railway-edge": "railway/asia-southeast1-eqsg3a",
"x-railway-request-id": "4D6qQA12Tl6_573HVOLIQQ",
"x-real-ip": "167.82.139.47",
"x-request-start": "1772161744984",
"x-timer": "S1772161745.944717,VS0",
"x-varnish": "3085425852"
},
"body": null
}from an echo server we deployed (result from a direct call)
applied these to test
Attachments
maybe, just maybe, the fastly headers are getting re-added at the upstream side (since the echo service is under a generated domain)
3 months ago
We are working on a fix that will strip fastly-ff on our Edge.
3 months ago
Update: Was planning to try stripping the fastly-ff header in our nginx gateway config (as suggested by @brody), but all our
custom domains just came back on their own (~7:56 PM PT). Looks like Railway resolved it on their end. All subdomains recovering
now.
3 months ago
Yep, see the above message. We now strip that header on our Edge.
3 months ago
Sorry about that, everyone, this was just one of those incredibly ultra-niche things you really have no way of knowing about beforehand.
Status changed to Solved brody • 3 months ago