5 months ago
Hi team,
We’re currently experiencing some abnormal behavior in our Railway infrastructure and wanted to report it.
-
Very slow deploys:
One of our frontend servers took more than 30 minutes to build, which is highly unusual for us. Our builds are typically much faster and we haven’t made significant changes that would justify such a delay.
-
Internal network instability:
After the build, we started seeing intermittent failures in the Railway internal network.
We have:
- A frontend being accessed via reverse proxy from another frontend
- Communication happening strictly over Railway’s internal network
This setup has been stable, but now requests occasionally fail between services.
- Backend database connectivity issues: One of our backend microservices (which connects to a MongoDB instance over Railway internal networking) started throwing database connection errors.
When attempting to redeploy the MongoDB service:
- It has been stuck in “Deploying” for more than 15 minutes
- Normally, this MongoDB service boots in ~5 seconds
This behavior is clearly outside our normal baseline.
Could you please check if there are:
- Ongoing infrastructure issues
- Internal networking degradation
- Regional cluster instability
- Or any incidents affecting deploy/build queues?
Thanks in advance.
Project ID: e60124ce-5bb4-41cb-8517-9606dbe5b9f7
Environment ID: 9700af47-ddfe-42ba-96b3-d3323dd60b27
MongoDB Service ID: d5274561-b8e2-4966-9c1c-1522fd37ce01
Attachments
18 Replies
Attachments
This guy tried to take me to another server, I think he should be banned.
5 months ago
Done
5 months ago
thx for the report
trying to redeploy again
Attachments
Now, magically, everything is working and the front-end server is no longer slow; everything seems to be flowing smoothly.
I created a replica in US-West for the frontend, could that have solved the problem?
5 months ago
Not the typical experience, I would say the unusual high platform demand was what affected you here. Railway gained a lot of new users in the last week or so and the team is actively working to make everything steady.
5 months ago
Not sure
5 months ago
Is everything working fine now?
I understand, this happened during an important deployment and caused some stress for the team, but everything seems to be fine now, thank you for your support!
hey, same happened here, our backend was unable to connect to our Mongo (hosted on Railway) for some reason. we fixed this by just redeploying both services, but this costed us 1 hour of downtime and is the second time in one week, we're worried
Looks like one of my services was randomly producing 502's up until I redeployed it just now, I see that there was an automatic deployment done by Railway about an hour ago, unsure if that's what caused it
two of my environments jsut stopped working, third time in a week! is anyone else still affected by the current incident?
5 months ago
Could you share a link to your service on Railway please?
the service is fine now but I was lucky enough for it to go back live after just one redeployment
5 months ago
Sorry about that - we had a transient routing issue that has since been resolved
Status changed to Solved brody • 5 months ago