2 years ago
During a Railway Incident, post any issues you have here and include as much detail as possible. Please do not create additional threads.
49 Replies
PS We're not on the new Edge Proxy and all our systems are down in Singapore. Perhaps this is the "TCP Proxy" purple bar? Just that the wording doesn't align
2 years ago
The only info I have is that the team is aware of and is working on the issue. Please stay tuned to <#846875565357006878> for more info
2 years ago
If you're online, sounds like they're making progress
2 years ago
That would be related to the incident. If you're still having issues after the incident is resolved, open a <#1006629907067064482> thread
2 years ago
We're seeing some recovery as we're rolling out the fix, still not 100% back.
2 years ago
glad to hear it, thanks rems
2 years ago
During the incident I saw that the network consumption skyrocketed, causing the charge to skyrocket as well, and the limit did not work.
2 years ago
That'll be something the team works out after the incident is resolved, but thanks for letting us know
My services all started working again, would love a post mortem on this since it downed my entire prod systme
2 years ago
same here. mongodb still down
2 years ago
( I like the service that Railway offers, but it is still a very immature company. There have been many errors lately.
I was thinking about going to PRO plan, but I don't trust.
I'm seriously considering migrating to DigitalOcean ) 🫤
Network usage skyrocketed and the usage limit did not work during the incident
2 years ago
same here. Thanks team! @railway
2 years ago
mysql service is still not accessible for me since the incident despite showing as online
2 years ago
The team is aware, you will be compensated
2 years ago
The incident is still active, your services being offline is expected
2 years ago
One of my services was available for 10 minutes and now not anymore
2 years ago
looks like it's working now. Hey Railway team, it may not be as simple as this but y'all should roll back to the legacy edge proxy that was working while you work on the fix. Once the fix is up deploy it. Downtime of an hour is critically bad and makes me consider going somewhere else
@Adam Will cost increases be taken into account on a case-by-case basis or how will the process be carried out?
2 years ago
that’s up to the team, I have no visibility into how they resolve cost issues
2 years ago
<#846875565357006878>
I accidentally deleted my old proxy and is it possible to see which one was old?
2 years ago
if your problem happened before the incident, please open a separate <#1006629907067064482> thread. Provide as much detail as possible
2 years ago
Are all the servers up again? because our server is still down
2 years ago
the team has reported that the proxies are nearly all back up. If you are still having issues, please post in here and give more details
2 years ago
<#846875565357006878>
2 years ago
Every region is now back up. We will be writing an incident report and sharing it by end of week (aiming tomorrow)
Apps using the tcp proxy for database connections might need a restart to kick back up and reconnect to the database(s). We are expecting some elevated latency as traffic is going back up, but there shouldn't be any major outage left.
If you're still running into issues now, I would invite you to create a new thread with the issue so we can take a quicker look at resolution. I will also be going through this thread
2 years ago
What extra information do you need? because our stage server is back up, but production isnt. Or does it just need a few more minutes?
2 years ago
How can we avoid this issues i future? I really love railway but it was never stable :/