a month ago
Hi the incident is still ongoing non of our services are running, multiple client app's are still down.
The number one issue now is that both our mongodb volume and postgress volume show 0 bytes (empty). we started a backup recovery on the mongodb volume but is has been running for a long time with no result, in the metrics it shows 1gb recovered from 8gb after 15 min.. and looks pretty much stuck now.
what can we do now? We need our services back up ASAP
I am getting pretty fed up with Railway and all the issues and outages in the last months.
4 Replies
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
Ok the restore finished but it restored only 1GB from nearly 8gb
where is my data? I need my services backup ASAP and my volumes restored!
Attachments
a month ago
No reply.... no help and other threads getting automatically closed.... good job Railway
Attachments
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
We apologize for the wait. The volume of support requests from today's incident has been exceptionally high, and we're doing our best to make sure everyone is looked at properly.
Looking at your Autodex project right now, all of your services are showing SUCCESS status, including MongoDB, Postgres, Redis, your worker, API, admin, CRM, and all frontends. It looks like you were able to get everything redeployed and running again. Regarding the volume restore that showed only ~1GB from the 7.64GB backup - your primary MongoDB volume with all your data is intact and attached to the running service. The partial restore created a separate volume from the backup snapshot, but your original data was never lost. Regardless, we apologize for how long the migration took to complete.
Please let us know if anything is still not working as expected.
Status changed to Awaiting Railway Response Railway • about 1 month ago
Status changed to Awaiting User Response Railway • about 1 month ago
Status changed to Solved solocommercenl • about 1 month ago