a year ago
During a Railway Incident, post any issues you have here and include as much detail as possible. Please do not create additional threads.
49 Replies
PS We're not on the new Edge Proxy and all our systems are down in Singapore. Perhaps this is the "TCP Proxy" purple bar? Just that the wording doesn't align
a year ago
The only info I have is that the team is aware of and is working on the issue. Please stay tuned to <#846875565357006878> for more info
a year ago
If you're online, sounds like they're making progress
a year ago
That would be related to the incident. If you're still having issues after the incident is resolved, open a <#1006629907067064482> thread
a year ago
We're seeing some recovery as we're rolling out the fix, still not 100% back.
a year ago
glad to hear it, thanks rems
a year ago
During the incident I saw that the network consumption skyrocketed, causing the charge to skyrocket as well, and the limit did not work.
a year ago
That'll be something the team works out after the incident is resolved, but thanks for letting us know
My services all started working again, would love a post mortem on this since it downed my entire prod systme
a year ago
same here. mongodb still down
a year ago
( I like the service that Railway offers, but it is still a very immature company. There have been many errors lately.
I was thinking about going to PRO plan, but I don't trust.
I'm seriously considering migrating to DigitalOcean ) 🫤
Network usage skyrocketed and the usage limit did not work during the incident
a year ago
same here. Thanks team! @railway
a year ago
mysql service is still not accessible for me since the incident despite showing as online
a year ago
The team is aware, you will be compensated
a year ago
The incident is still active, your services being offline is expected
a year ago
One of my services was available for 10 minutes and now not anymore
a year ago
looks like it's working now. Hey Railway team, it may not be as simple as this but y'all should roll back to the legacy edge proxy that was working while you work on the fix. Once the fix is up deploy it. Downtime of an hour is critically bad and makes me consider going somewhere else
@Adam Will cost increases be taken into account on a case-by-case basis or how will the process be carried out?
a year ago
that’s up to the team, I have no visibility into how they resolve cost issues
a year ago
<#846875565357006878>
I accidentally deleted my old proxy and is it possible to see which one was old?
a year ago
if your problem happened before the incident, please open a separate <#1006629907067064482> thread. Provide as much detail as possible
a year ago
Are all the servers up again? because our server is still down
a year ago
the team has reported that the proxies are nearly all back up. If you are still having issues, please post in here and give more details
a year ago
<#846875565357006878>
a year ago
Every region is now back up. We will be writing an incident report and sharing it by end of week (aiming tomorrow)
Apps using the tcp proxy for database connections might need a restart to kick back up and reconnect to the database(s). We are expecting some elevated latency as traffic is going back up, but there shouldn't be any major outage left.
If you're still running into issues now, I would invite you to create a new thread with the issue so we can take a quicker look at resolution. I will also be going through this thread
a year ago
What extra information do you need? because our stage server is back up, but production isnt. Or does it just need a few more minutes?
a year ago
How can we avoid this issues i future? I really love railway but it was never stable :/