Railway-initiated deployment: Volume failed migration

okarmazin
PRO

a month ago

https://railway.com/project/8093f881-dd83-4250-b725-dd87c94a218e/volume/948c1bba-9be2-4729-973d-18c2c72d610d/metrics?environmentId=cdd73b5e-4d2d-4c8d-9bec-1d90c10619bd

Hello, the linked volume is in an error state following Railway-initiated redeployment of the database service. Triggering another service deploy manually doesn't help, I believe a Railway team member will be needed here.

Solved

14 Replies

btnalexandre
PRO

a month ago

Hello same problem for me


philey055
HOBBY

a month ago

same here..


callmefredcom
PRO

a month ago

Same for me. Why those Railway-initiated redeployments? Why do they fail?


jack-hannington
PRO

a month ago

Same for me, redeploying has worked but not great to have it go down unexpectedly.


a month ago

Same for me. Volume was already migrated to new region (if that's the reason of this automatic trigger) but results in "Volume migration failed". I did a "retry deployment" and it went through succesfully.

Now I have a second volume trying to migrate with "Waiting to migrate to new region.", which is already in the Metal/europe new region, it's been more than 60minutes, for ~150MB, and still "waiting to migrate"


a month ago

I’ve also had two Postgres deployments fail.


niko
PRO

a month ago

Same for me, it also happened on a Qdrant instance (as for a postgres instance, and 2 different environment with a Postgres on each of them)


richmeij
PRO

a month ago

Same for (some of) our deployments.


raphaelflash
HOBBY

a month ago

I had the same issue, but clicking redeploy manually seems to resolve it.


arnorhs
PRO

a month ago

same here.. is there some info about this somewhere? was this a scheduled thing that I wasn't aware of?


a month ago

!t


a month ago

This thread has been escalated to the Railway team.

Status changed to Awaiting Railway Response uxuz about 1 month ago


arnorhs
PRO

a month ago

also.. it's pretty weird to have the message

Volume migration failed.

There was an error migrating your volume to the new region.

when i wasn't aware that my volumes were being moved to a new region.. which region is it going to?


arnorhs
PRO

a month ago

is it related to migrating to metal? my volumes were already on metal..


arnorhs
PRO

a month ago

redeploying didn't resolve the volume migration error btw..


okarmazin
PRO

a month ago

That's just a generic error message. They are likely using the same mechanisms for this as region to region moves.


namrastanov
HOBBY

a month ago

Same for me. I got random database deployment which was failed and no logs. But the app works as expected.


grbeno
HOBBY

a month ago

Same issue. My case: In two of my projects, postgres and redis are affected. App works, but I can't reach the tables in Postgres/Data on the dashboard.


positive-endeavour
PRO

a month ago

I have the same issue with the volume migration failing.
My django app appears to be working. I am a little inexperienced with deployment, so just wondering if there is anything I need to do?


a month ago

Hello,

okarmazin, I have raised your migration failure to the team.

For everyone else, please provide direct links to the service that has a failed migration.


arnorhs
PRO

a month ago

@Brody to each service? in all environments? basically for me, it's every service in every environment that has an attached volume


arnorhs
PRO

a month ago

actually, one of them didn't fail.. so it's 5 services in total, should I link them all?


a month ago

yes please








sdcooke
PRO

a month ago

This also happened to me - both of my services have now been redeployed in the last two days (one of them failed the first time) and are working fine, but now the observability graph shows double the data usage for each of the two volumes, which has triggered my monitors. Is it possible that the volumes are now being double-tracked?



arnorhs
PRO

a month ago

is there an updated status on this issue?


a month ago

Hey there everyone,

So update, there are a variety of reasons why this would report the way it does and it's hard to exfil that information publicly.

However, what I can say is that we are working on a fix where if a Railway initiated migration fails (move your stuff to Metal!) it will not show as a red failure on your project. If you need a 1 to 1 manual migration from the Railway team, I would make a private thread with the team and we can work with you there.


Status changed to Awaiting User Response Railway about 1 month ago


a month ago

On my side, I already migrated everything to Metal 2 or 3 months ago.

However in the past few days, I got 4 or 5 automatic deployments from railway telling me my services were being migrated to metal. Either volumes (where migration failed but was already on metal) or services (reporting successful metal deployment but was already in metal prior).


okarmazin
PRO

a month ago

I think everybody in this thread has already moved to Metal months ago.


arnorhs
PRO

a month ago

yes, moved to metal months ago, but my postgres volume still has a volume migration failed sticker on it.. is it safe to redeploy the service and will that resolve the volume migration failed error?


arnorhs
PRO

a month ago

without data loss 😄



Status changed to Awaiting Railway Response Railway about 1 month ago


Status changed to Awaiting User Response chandrika about 1 month ago


adelelsawaf

https://railway.com/project/69b7bbfd-c33f-4ea5-99b8-412c134a04f7/service/26fa8933-edde-47ee-8048-9a4951a83a9e/settings?environmentId=fb618fa5-dc92-47ac-b52b-e585aaee0273same problem for this service

chandrika
EMPLOYEE

a month ago

Hi there, I've responded on your private thread with us — you should be all set now


a month ago

!s


Status changed to Solved brody 27 days ago


Railway-initiated deployment: Volume failed migration - Railway Help Station