Volume migration failed
maksspace
PROOP

7 months ago

Hi, couple of hours ago database deployment crashed. And I can not deploy it, because every time this error pops out.

https://railway.com/project/e1026f82-be00-46c0-b213-850a7b6f4537/service/5616022a-64b9-4e92-b018-a41b543d3a0b/data?environmentId=ec556211-a9bb-437d-b3ea-a5351246fbfd

Attachments

Solved

20 Replies

Railway
BOT

7 months ago

Hey there! We've found the following might help you get unblocked faster:

If you find the answer from one of these, please let us know by solving the thread!


maksspace
PROOP

7 months ago

some other error

Attachments


Railway

Hey there! We've found the following might help you get unblocked faster: - [🧵 Volume migration failed](https://station.railway.com/questions/volume-migration-failed-8d8619cc) - [🧵 Railway-initiated deployment: Volume failed migration](https://station.railway.com/questions/railway-initiated-deployment-volume-fai-e10224c5) - [🧵 Volume Migration failed](https://station.railway.com/questions/volume-migration-failed-0002122d) If you find the answer from one of these, please let us know by solving the thread!

maksspace
PROOP

7 months ago

I’ve already seen those issues, and from what I can tell, many of them were resolved by the Railway team.


maksspace
PROOP

7 months ago

it's URGENT!!!!!!!!!


Railway
BOT

7 months ago

Hello!

We've escalated your issue to our engineering team.

We aim to provide an update within 1 business day.

Please reply to this thread if you have any questions!

Status changed to Awaiting User Response Railway 7 months ago


Hey Maks, we're looking into this issue.


Okay, you should be back. We're running an investigation as to why this occured.


aderatis
HOBBY

7 months ago

sorry to jump on this thread too, I am having the same issue, and this is a critical live business application, database vanished for no reason, tried to deploy it again, after hours and hours it failed multiple times, help!!: https://railway.com/project/63d1d05d-71e3-44f4-917b-2e7e71305b09/volume/1fded76d-75a1-48af-8d82-99dcee6ff7be/settings?environmentId=4626f72e-a427-4c12-84c6-909f6a75b119


Status changed to Awaiting Railway Response Railway 7 months ago


aderatis

sorry to jump on this thread too, I am having the same issue, and this is a critical live business application, database vanished for no reason, tried to deploy it again, after hours and hours it failed multiple times, help!!: https://railway.com/project/63d1d05d-71e3-44f4-917b-2e7e71305b09/volume/1fded76d-75a1-48af-8d82-99dcee6ff7be/settings?environmentId=4626f72e-a427-4c12-84c6-909f6a75b119

No issue at all, we want everyone to be in a good spot.


Status changed to Awaiting User Response Railway 7 months ago


I have kicked a new deployment for you. We found the cause of the issue in question, one of the nodes we use for orchestration went offline and didn't alert us. We have since fixed the issue and added more alerting on that component. First time this happened, and hopefully last. It's an extremely scary bug to see.


angelo-railway

I have kicked a new deployment for you. We found the cause of the issue in question, one of the nodes we use for orchestration went offline and didn't alert us. We have since fixed the issue and added more alerting on that component. First time this happened, and hopefully last. It's an extremely scary bug to see.

aderatis
HOBBY

7 months ago

that's great thank you


Status changed to Awaiting Railway Response Railway 7 months ago


aderatis

that's great thank you

.....and you're back.


Status changed to Awaiting User Response Railway 7 months ago


angelo-railway

.....and you're back.

a7i7bek
HOBBY

7 months ago

I had the same issue a couple of hours ago. Now I am having another issue: "Failed to create deployment." Does it relate to the Volume migration? Because, I did not have any problem with deployments up until now. Please, help pray emoji

https://railway.com/project/3e9f09e0-8736-4be8-abbe-8ac13c817d08/service/ea39924b-cf8e-4f35-a1b0-708787a978ab?environmentId=fafdaf00-aade-4084-b166-b4c50e0daa52


Status changed to Awaiting Railway Response Railway 7 months ago


angelo-railway

I have kicked a new deployment for you. We found the cause of the issue in question, one of the nodes we use for orchestration went offline and didn't alert us. We have since fixed the issue and added more alerting on that component. First time this happened, and hopefully last. It's an extremely scary bug to see.

aderatis
HOBBY

7 months ago

Hi Angelo, I had to revert some changes in the site using Cursor, it has deployed and used the new database url you generated but the site isn't functioning, I have asked it to audit why that is and it's outputted the following:
hammer_and_wrench emoji What's Happening:

  1. Your site IS working - Django is connecting and updating sessions

  1. Railway is killing connections - probably due to their database migration

  1. This causes HTTP 502 errors - because connections keep getting killed

clipboard emoji Railway Support Needed:

This is NOT something I can fix in your code. Railway needs to:

  1. Stop killing database connections

  1. Complete their database migration properly

  1. Stabilize the database service

What You Need to Do:

Contact Railway Support:

  • Tell them: "Database connections are being terminated with 'FATAL: terminating connection due to administrator command'"

  • Ask them to complete the database migration properly

  • Request they stop killing active connections

Your Code is Fine:

  • Django is working (updating sessions successfully)

  • The database is accessible

  • Railway is the problem, not your code

question emoji Next Steps:

  1. Contact Railway support about the connection terminations

  1. Wait for them to stabilize the database service

  1. Your site will work once they stop killing connections

This is 100% a Railway platform issue that they need to resolve. Your code is working perfectly!

Does this sound feasible? this issue is giving me headache lol.


a7i7bek

I had the same issue a couple of hours ago. Now I am having another issue: "Failed to create deployment." Does it relate to the Volume migration? Because, I did not have any problem with deployments up until now. Please, help https://railway.com/project/3e9f09e0-8736-4be8-abbe-8ac13c817d08/service/ea39924b-cf8e-4f35-a1b0-708787a978ab?environmentId=fafdaf00-aade-4084-b166-b4c50e0daa52

Acking. This is unrelated to the root, but I have escalated it all the same. Appreciate your patience.


Status changed to Awaiting User Response Railway 7 months ago


aderatis
HOBBY

7 months ago

Hi Angelo, I suspect it's cursor being a pita. I have managed to get the site back live now thank you, i just need to try figure out the coding errors to make it work as it should. Do I have backup recourse in place if this ever happens again on your end with the node problem?


Status changed to Awaiting Railway Response Railway 7 months ago


aderatis

Hi Angelo, I suspect it's cursor being a pita. I have managed to get the site back live now thank you, i just need to try figure out the coding errors to make it work as it should. Do I have backup recourse in place if this ever happens again on your end with the node problem?

We can't control that lol.

Yes, after review, this is not related to the OP issue. The problem here is that we had a bug where Railway would just "forget" the volume location, there's a few customers that need to manually nudge into place. After nudging it, it works. The DB issue you faced looked like a application issue, now resolved.


Status changed to Awaiting User Response Railway 7 months ago


gamehdr103
HOBBY

7 months ago

J'ai le meme problème avec mon projet C'est Urgent il y a des projets en production


Status changed to Awaiting Railway Response Railway 7 months ago


gamehdr103

J'ai le meme problème avec mon projet C'est Urgent il y a des projets en production

gamehdr103
HOBBY

7 months ago

I redeployed it's happening


OP is solved and I see everything working on your side? Am I good to solve this thread?


Status changed to Awaiting User Response Railway 7 months ago


Status changed to Solved ray-chen 6 months ago


Railway
BOT

6 months ago

✅ The ticket Storage migration issue has been marked as completed.


Loading...