22 days ago
Hi team,
We’re currently experiencing some abnormal behavior in our Railway infrastructure and wanted to report it.
Very slow deploys:
One of our frontend servers took more than 30 minutes to build, which is highly unusual for us. Our builds are typically much faster and we haven’t made significant changes that would justify such a delay.Internal network instability:
After the build, we started seeing intermittent failures in the Railway internal network.
We have:
A frontend being accessed via reverse proxy from another frontend
Communication happening strictly over Railway’s internal network
This setup has been stable, but now requests occasionally fail between services.
Backend database connectivity issues:
One of our backend microservices (which connects to a MongoDB instance over Railway internal networking) started throwing database connection errors.
When attempting to redeploy the MongoDB service:
It has been stuck in “Deploying” for more than 15 minutes
Normally, this MongoDB service boots in ~5 seconds
This behavior is clearly outside our normal baseline.
Could you please check if there are:
Ongoing infrastructure issues
Internal networking degradation
Regional cluster instability
Or any incidents affecting deploy/build queues?
Thanks in advance.
Project ID: e60124ce-5bb4-41cb-8517-9606dbe5b9f7
Environment ID: 9700af47-ddfe-42ba-96b3-d3323dd60b27
MongoDB Service ID: d5274561-b8e2-4966-9c1c-1522fd37ce01
Attachments
18 Replies
Attachments
This guy tried to take me to another server, I think he should be banned.
22 days ago
Done
22 days ago
thx for the report
trying to redeploy again
Attachments
Now, magically, everything is working and the front-end server is no longer slow; everything seems to be flowing smoothly.
I created a replica in US-West for the frontend, could that have solved the problem?
22 days ago
Not the typical experience, I would say the unusual high platform demand was what affected you here. Railway gained a lot of new users in the last week or so and the team is actively working to make everything steady.
22 days ago
Not sure
22 days ago
Is everything working fine now?
I understand, this happened during an important deployment and caused some stress for the team, but everything seems to be fine now, thank you for your support!
hey, same happened here, our backend was unable to connect to our Mongo (hosted on Railway) for some reason. we fixed this by just redeploying both services, but this costed us 1 hour of downtime and is the second time in one week, we're worried
Looks like one of my services was randomly producing 502's up until I redeployed it just now, I see that there was an automatic deployment done by Railway about an hour ago, unsure if that's what caused it
two of my environments jsut stopped working, third time in a week! is anyone else still affected by the current incident?
22 days ago
Could you share a link to your service on Railway please?
the service is fine now but I was lucky enough for it to go back live after just one redeployment
https://railway.com/project/204594ed-2cbb-4f9e-924a-88cf5d199e27/service/780b1dcc-92b7-43f3-9b40-f71f5299894b?environmentId=767ba06f-5057-4e54-9c14-7104bd5ae0e2
22 days ago
Sorry about that - we had a transient routing issue that has since been resolved
Status changed to Solved brody • 21 days ago