a764e147-c28b-4388-85d2-547de92d2fd0

Can anyone help? This affects our production server.

I could not even restore from backup

It keeps going on even after 25 minutes which I cancelled and retried

This is a good reminder not to use a single mongodb service on prod services where you need high reliability and instead use the distributed template instead to handle these cases where your primary is offline.

How much storage do you have, do you see it progressing the migration when it runs?

50GB allocated, but only 1.312GB used
How do I see the progress of the migration? It only says "Waiting for volume migration"

I haven't used railway for a month or two - but iirc there should be a Migration progress bar below your volume stating the percentage progress.

I'll leave this to someone else to answer the question, since I've not touched migrations in a lot of depth since they were introduced - I ended up moving to self-hosting when they introduced cloud.

This is a good opportunity for me to plug https://github.com/PostSuite/railway-derailer to test your systems for reliability if something is to go offline.

Not sure if is related only to you, the same is happening to postgress

please open your own thread and link to the service that is experiencing this issue.

@Celengan Babi - I can't see any services where this is still happening. Can you please link me the service?

Hi @Brody, here's the link:
https://railway.com/project/a764e147-c28b-4388-85d2-547de92d2fd0/service/b9fb0dee-37c8-4ac1-8950-c1bad2a928e6/data?environmentId=81c89206-a39a-49da-8c57-eadb333b1784

thank you! raising to the team now

It just failed, so maybe that's the reason why it doesn't appear on your end.

In the meantime, I've instantiated another MongoDB service and restored the backup. So everything is fine for now.
But it would be great to be able to know the exact reason why that service was even "automatically redeployed" by Railway…

The service was on Metal US East region already, it was migrated.

So there shouldn't be any reason for the redeployment…

Even if there should be any redeployment, why was it stuck in "Creating containers" step?
This could have gone worse had I gone to sleep already 😦

as for why it deployed -

https://docs.railway.com/reference/deployments#railway-initiated-deployments

as for why it failed to deploy, i have raised that to the team and i will let you know as soon as i know.

i'm really sorry about the downtime this has casued.

Thank you. Hope this gets resolved soon and won't happen again in the future 🙏

Following up here.

We've identified that a component in our deployment infrastructure became unresponsive, which caused your deployment to hang before eventually failing. We sincerely apologize for the frustration this caused.

We've now implemented additional monitoring to detect this type of issue immediately, so our team can resolve it much faster if it happens again.

Thanks for your patience, and please don't hesitate to reach out again if you have any other questions or concerns.

Also, I wanted to clarify our support response times to set proper expectations going forward.

As a Pro user, our standard response time is within 24 hours, which we maintained for your ticket. For customers who need faster response times (within 1 hour for critical issues), we offer that level of support through our Enterprise plan.

If faster response times would be valuable for your use case, I'd be happy to connect you with our sales team to discuss Enterprise options.

Thanks again for your understanding!

Thank you, Brody. Much appreciated 🙏

!s