a month ago
I had deployed everything and all of that is gone, after the outage, how do I recover all of that? Do I need to redeploy everything?!
14 Replies
a month ago
Your services (MongoDB and Postgres in the "svantic" project) still exist and both show a successful deployment status. Nothing has been deleted. The recent service disruption is now in the monitoring phase, and services are recovering. If your services aren't responding correctly, try triggering a redeploy from the dashboard or CLI.
Status changed to Awaiting User Response Railway • about 1 month ago
Railway
Your services (MongoDB and Postgres in the "svantic" project) still exist and both show a successful deployment status. Nothing has been deleted. The recent [service disruption](https://status.railway.com/incident/I23M92U0) is now in the monitoring phase, and services are recovering. If your services aren't responding correctly, try triggering a redeploy from the dashboard or CLI.
a month ago
The entire configuraiton is gone though - I had replica sets, etc running earlier. I had services configured. how do I retrigger that?
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
We sincerely apologize for the service disruption. Your services and project still exist - nothing was deleted. The disruption caused running workloads to stop, which is why your replica sets and configuration appear gone.
We're seeing recovery across our API, builds, and deployments. Please try triggering a redeploy from the dashboard or CLI for each affected service to restore them to their previous running state. I manually triggered a deploy of Mongo for you and it has come up without issue now.
For ongoing context while we work on a postmortem, you can read our FAQ here.
Status changed to Awaiting User Response Railway • about 1 month ago
Status changed to Solved mykal • about 1 month ago
sapshah
The entire configuraiton is gone though - I had replica sets, etc running earlier. I had services configured. how do I retrigger that?
a month ago
I am not sure if you understand. There is no way I can trigger a redeploy. You are asking me to reconfigure my entire infrastructure - it was a git managed pipeline - that is gone. The services I had added are gone. The repositiories I had linked are gone. Are you saying that I will have to cnofigure all of this again?
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
Hey Sapshah, apologies for the back and forth. Can you share anything more about what happened? All of your services were connected and configured and all those settings were reset? I'm seeing standalone services but no connections between them and see recent commits trying to correct the start command to probably reconcile the project back to what it was?
Status changed to Awaiting User Response Railway • about 1 month ago
a month ago
When I was told that deployments were working again, I logged into Railway only to find that none of my configured services were present. I had multiple GitHub repositories connected, and all of those integrations were gone as well.
After some time, the nodes eventually reappeared, but the GitHub connectivity had been completely lost. As a result, I had to reconfigure everything from scratch, and I’m still in the process of recovering the environment.
At the moment, I’m also unable to remap the custom domains that were previously configured for these services. Railway keeps showing “domain is not available” whenever I try to add them back.
On top of that, I’m unable to access logs for the affected services, which is making troubleshooting significantly more difficult.
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
Interestingly enough, there is a copy of this running on railway because I am able to access that somehow. So somewhere there are two infras runnning and its causing issue sbecause I dont knwo which one is waht
a month ago
any update on this? its just not working for me? I am afraid this cretes a lot of issues since parts of services are working and parts are not. I am unable to deploy the new setup
chandrika
Hey Sapshah, apologies for the back and forth. Can you share anything more about what happened? All of your services were connected and configured and all those settings were reset? I'm seeing standalone services but no connections between them and see recent commits trying to correct the start command to probably reconcile the project back to what it was?
a month ago
When I was told that deployments were working again, I logged into Railway only to find that none of my configured services were present. I had multiple GitHub repositories connected, and all of those integrations were gone as well.
After some time, the nodes eventually reappeared, but the GitHub connectivity had been completely lost. As a result, I had to reconfigure everything from scratch, and I’m still in the process of recovering the environment.
At the moment, I’m also unable to remap the custom domains that were previously configured for these services. Railway keeps showing “domain is not available” whenever I try to add them back.
On top of that, I’m unable to access logs for the affected services, which is making troubleshooting significantly more difficult.
sapshah
When I was told that deployments were working again, I logged into Railway only to find that none of my configured services were present. I had multiple GitHub repositories connected, and all of those integrations were gone as well. After some time, the nodes eventually reappeared, but the GitHub connectivity had been completely lost. As a result, I had to reconfigure everything from scratch, and I’m still in the process of recovering the environment. At the moment, I’m also unable to remap the custom domains that were previously configured for these services. Railway keeps showing “domain is not available” whenever I try to add them back. On top of that, I’m unable to access logs for the affected services, which is making troubleshooting significantly more difficult.
a month ago
any update on this? its just not working for me? I am afraid this cretes a lot of issues since parts of services are working and parts are not. I am unable to deploy the new setup
a month ago
I haven't heard back yet. I am awaiting response, and this is a blocker for us.
chandrika
Hey Sapshah, apologies for the back and forth. Can you share anything more about what happened? All of your services were connected and configured and all those settings were reset? I'm seeing standalone services but no connections between them and see recent commits trying to correct the start command to probably reconcile the project back to what it was?
a month ago
Can I please get an update on this? I am afaried I will need to move away from railway in this case. I am trying to recreate it, but it wont work.
sapshah
Can I please get an update on this? I am afaried I will need to move away from railway in this case. I am trying to recreate it, but it wont work.
a month ago
Just to reiterate - after the last outage, I am unable to remap my domains; as a result, the hit on the domains leads me to a phantom old deployment, which should not happen.
sapshah
Just to reiterate - after the last outage, I am unable to remap my domains; as a result, the hit on the domains leads me to a phantom old deployment, which should not happen.
a month ago
On checking with Railway agent I got this: You need to contact Railway support directly. This requires manual intervention to:
Clear the orphaned domain registrations from the outage
Release api.svantic.com and app.svantic.com so they can be re-claimed
When you reach out, provide:
Project ID: 3437ba2f-04d6-4372-8b62-368efa4b458e
Environment: production
Affected domains: api.svantic.com, app.svantic.com
Issue: Domains marked "not available" after outage; need manual DNS cleanup to restore routing
The domains are already configured in the api-edge service config (I've staged that), but the UI block is a platform-level issue only support can resolve.