4 months ago
I've opened up a support ticket but its been a day with no response so creating a public thread hoping someone might know whats going on.
My node API is randomly going down return 502 errors. No change has been made on my end. Only a rollback (not a redeploy) seems to resolve the issue temporarily.
The generated public domain provided by railway works, the custom domain is what throws the below error. That domain is managed by cloudflare, but nothing has changed on my end for months.
{
"status": "error",
"code": 502,
"message": "Application failed to respond",
"request_id": "pqo6l4Z8TYeGHtzGueLfZA"
}.7 Replies
4 months ago
Correction, I tried again using the railway generated domain name and it returns a 502, this removes cloudflare from the picture.
thoggard
Correction, I tried again using the railway generated domain name and it returns a 502, this removes cloudflare from the picture.
4 months ago
Hey thoggard, can you confirm that your node api is not under heavy load?
For example, if you are running a Node.js application, and see that your vCPU usage has peaked at any point to around 1 vCPU, this is a good indication that your application is under heavy load given Node's single-threaded nature.
Docs: https://docs.railway.com/reference/errors/application-failed-to-respond#application-under-heavy-load
4 months ago
Hey uxuz, the max load is about 0.1 vCPU and the server is configured to have a Max of 32 vCPU. Memory is also good. I cloned my server and was able to reproduce the issue with zero load. When I attempt to hit the server it logs a single HTTP log (nothing logged by my server, I log every request)
requestId:"HGzx0g9NQVWX0jgQueLfZA"
timestamp:"2025-05-10T21:48:51.581059290Z"
method:"GET"
path:"/"
host:"redacted"
httpStatus:502
upstreamProto:"HTTP/1.1"
downstreamProto:"HTTP/2.0"
responseDetails:"failed to forward request to upstream: connection refused"
totalDuration:155
upstreamAddress:"redacted"
clientUa:"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"
upstreamRqDuration:93
txBytes:4682
rxBytes:754
srcIp:"134.231.60.210"
edgeRegion:"us-east4"
4 months ago
I updated the cloned server so it uses the same commit the main server uses. The same commit works on the main server, but does not work on the cloned server.
thoggard
I updated the cloned server so it uses the same commit the main server uses. The same commit works on the main server, but does not work on the cloned server.
4 months ago
In conclusion, your service respond 502 on random occasions, which is also reproducible by duplicating the service in question with identical configuration. Am I getting this right?
I still have two questions. The first being when this issue originally started or has this always been an issue since you have deployed the service? Additionally, is the same behavior also present when interacting with your service via private networking?
(I'll go to bed now and will get back to you at a later time.)
uxuz
In conclusion, your service respond 502 on random occasions, which is also reproducible by duplicating the service in question with identical configuration. Am I getting this right?I still have two questions. The first being when this issue originally started or has this always been an issue since you have deployed the service? Additionally, is the same behavior also present when interacting with your service via private networking?(I'll go to bed now and will get back to you at a later time.)
4 months ago
Almost, the 502 error happens 100% of the time on any new deploys. The only way to fix this is to rollback, redeploying doesn't work. The configurations are identical between the main server and the cloned one, same secrets and git commit.
This issue only started happening yesterday, I did a deploy the day before without any issues. This is the first time I'm seeing this over the past 2 years I've been using railway.
I'm not sure how to test the private networking stuff. I tried using the railway CLI and ssh into the container and then run curl on the private URL but says "Coudn't connect to server". Is there a better way?
4 months ago
I fixed the issue, turns out that I was using ::1 but railway expected :: for the servers host IP. Now, why all of a sudden this started causing a 502 I have no idea, its been configured like this for 2 years. I have a feeling there was a docker configuration change made by railway that exposed this issue.
Status changed to Solved brody • 4 months ago