Railway-initiated deployment: Volume failed migration
okarmazin
PROOP

8 months ago

https://railway.com/project/8093f881-dd83-4250-b725-dd87c94a218e/volume/948c1bba-9be2-4729-973d-18c2c72d610d/metrics?environmentId=cdd73b5e-4d2d-4c8d-9bec-1d90c10619bd

Hello, the linked volume is in an error state following Railway-initiated redeployment of the database service. Triggering another service deploy manually doesn't help, I believe a Railway team member will be needed here.

Solved

39 Replies

btnalexandre
PRO

8 months ago

Hello same problem for me confused emoji


philey055
PRO

8 months ago

same here..


callmefredcom
PRO

8 months ago

Same for me. Why those Railway-initiated redeployments? Why do they fail?


jack-hannington
PRO

8 months ago

Same for me, redeploying has worked but not great to have it go down unexpectedly.


8 months ago

Same for me. Volume was already migrated to new region (if that's the reason of this automatic trigger) but results in "Volume migration failed". I did a "retry deployment" and it went through succesfully.

Now I have a second volume trying to migrate with "Waiting to migrate to new region.", which is already in the Metal/europe new region, it's been more than 60minutes, for ~150MB, and still "waiting to migrate"


8 months ago

I’ve also had two Postgres deployments fail.


niko
PRO

8 months ago

Same for me, it also happened on a Qdrant instance (as for a postgres instance, and 2 different environment with a Postgres on each of them)


richmeij
PRO

8 months ago

Same for (some of) our deployments.


raphaelflash
PRO

8 months ago

I had the same issue, but clicking redeploy manually seems to resolve it.


arnorhs
PRO

8 months ago

same here.. is there some info about this somewhere? was this a scheduled thing that I wasn't aware of?


8 months ago

!t


8 months ago

This thread has been escalated to the Railway team.

Status changed to Awaiting Railway Response uxuz 8 months ago


arnorhs
PRO

8 months ago

also.. it's pretty weird to have the message

Volume migration failed.

There was an error migrating your volume to the new region.

when i wasn't aware that my volumes were being moved to a new region.. which region is it going to?


arnorhs
PRO

8 months ago

is it related to migrating to metal? my volumes were already on metal..


arnorhs
PRO

8 months ago

redeploying didn't resolve the volume migration error btw..


okarmazin
PROOP

8 months ago

That's just a generic error message. They are likely using the same mechanisms for this as region to region moves.


namrastanov
HOBBY

8 months ago

Same for me. I got random database deployment which was failed and no logs. But the app works as expected.


grbeno
HOBBY

8 months ago

Same issue. My case: In two of my projects, postgres and redis are affected. App works, but I can't reach the tables in Postgres/Data on the dashboard.


positive-endeavour
PRO

8 months ago

I have the same issue with the volume migration failing.
My django app appears to be working. I am a little inexperienced with deployment, so just wondering if there is anything I need to do?


8 months ago

Hello,

okarmazin, I have raised your migration failure to the team.

For everyone else, please provide direct links to the service that has a failed migration.


arnorhs
PRO

8 months ago

@Brody to each service? in all environments? basically for me, it's every service in every environment that has an attached volume


arnorhs
PRO

8 months ago

actually, one of them didn't fail.. so it's 5 services in total, should I link them all?


8 months ago

yes please








sdcooke
PRO

8 months ago

This also happened to me - both of my services have now been redeployed in the last two days (one of them failed the first time) and are working fine, but now the observability graph shows double the data usage for each of the two volumes, which has triggered my monitors. Is it possible that the volumes are now being double-tracked?



arnorhs
PRO

8 months ago

is there an updated status on this issue?


Hey there everyone,

So update, there are a variety of reasons why this would report the way it does and it's hard to exfil that information publicly.

However, what I can say is that we are working on a fix where if a Railway initiated migration fails (move your stuff to Metal!) it will not show as a red failure on your project. If you need a 1 to 1 manual migration from the Railway team, I would make a private thread with the team and we can work with you there.


Status changed to Awaiting User Response Railway 8 months ago


8 months ago

On my side, I already migrated everything to Metal 2 or 3 months ago.

However in the past few days, I got 4 or 5 automatic deployments from railway telling me my services were being migrated to metal. Either volumes (where migration failed but was already on metal) or services (reporting successful metal deployment but was already in metal prior).


okarmazin
PROOP

8 months ago

I think everybody in this thread has already moved to Metal months ago.


arnorhs
PRO

8 months ago

yes, moved to metal months ago, but my postgres volume still has a volume migration failed sticker on it.. is it safe to redeploy the service and will that resolve the volume migration failed error?


arnorhs
PRO

8 months ago

without data loss 😄



Status changed to Awaiting Railway Response Railway 8 months ago


Status changed to Awaiting User Response chandrika 8 months ago


adelelsawaf

https://railway.com/project/69b7bbfd-c33f-4ea5-99b8-412c134a04f7/service/26fa8933-edde-47ee-8048-9a4951a83a9e/settings?environmentId=fb618fa5-dc92-47ac-b52b-e585aaee0273same problem for this service

chandrika
EMPLOYEE

8 months ago

Hi there, I've responded on your private thread with us — you should be all set now raised_hands emoji


8 months ago

!s


Status changed to Solved brody 8 months ago


Loading...