7 months ago
Hi, couple of hours ago database deployment crashed. And I can not deploy it, because every time this error pops out.
Attachments
20 Replies
7 months ago
Hey there! We've found the following might help you get unblocked faster:
If you find the answer from one of these, please let us know by solving the thread!
Railway
Hey there! We've found the following might help you get unblocked faster: - [🧵 Volume migration failed](https://station.railway.com/questions/volume-migration-failed-8d8619cc) - [🧵 Railway-initiated deployment: Volume failed migration](https://station.railway.com/questions/railway-initiated-deployment-volume-fai-e10224c5) - [🧵 Volume Migration failed](https://station.railway.com/questions/volume-migration-failed-0002122d) If you find the answer from one of these, please let us know by solving the thread!
7 months ago
I’ve already seen those issues, and from what I can tell, many of them were resolved by the Railway team.
7 months ago
it's URGENT!!!!!!!!!
7 months ago
Hello!
We've escalated your issue to our engineering team.
We aim to provide an update within 1 business day.
Please reply to this thread if you have any questions!
Status changed to Awaiting User Response Railway • 7 months ago
7 months ago
Okay, you should be back. We're running an investigation as to why this occured.
7 months ago
sorry to jump on this thread too, I am having the same issue, and this is a critical live business application, database vanished for no reason, tried to deploy it again, after hours and hours it failed multiple times, help!!: https://railway.com/project/63d1d05d-71e3-44f4-917b-2e7e71305b09/volume/1fded76d-75a1-48af-8d82-99dcee6ff7be/settings?environmentId=4626f72e-a427-4c12-84c6-909f6a75b119
Status changed to Awaiting Railway Response Railway • 7 months ago
aderatis
sorry to jump on this thread too, I am having the same issue, and this is a critical live business application, database vanished for no reason, tried to deploy it again, after hours and hours it failed multiple times, help!!: https://railway.com/project/63d1d05d-71e3-44f4-917b-2e7e71305b09/volume/1fded76d-75a1-48af-8d82-99dcee6ff7be/settings?environmentId=4626f72e-a427-4c12-84c6-909f6a75b119
7 months ago
No issue at all, we want everyone to be in a good spot.
Status changed to Awaiting User Response Railway • 7 months ago
7 months ago
I have kicked a new deployment for you. We found the cause of the issue in question, one of the nodes we use for orchestration went offline and didn't alert us. We have since fixed the issue and added more alerting on that component. First time this happened, and hopefully last. It's an extremely scary bug to see.
angelo-railway
I have kicked a new deployment for you. We found the cause of the issue in question, one of the nodes we use for orchestration went offline and didn't alert us. We have since fixed the issue and added more alerting on that component. First time this happened, and hopefully last. It's an extremely scary bug to see.
7 months ago
that's great thank you
Status changed to Awaiting Railway Response Railway • 7 months ago
Status changed to Awaiting User Response Railway • 7 months ago
angelo-railway
.....and you're back.
7 months ago
I had the same issue a couple of hours ago. Now I am having another issue: "Failed to create deployment." Does it relate to the Volume migration? Because, I did not have any problem with deployments up until now. Please, help 
https://railway.com/project/3e9f09e0-8736-4be8-abbe-8ac13c817d08/service/ea39924b-cf8e-4f35-a1b0-708787a978ab?environmentId=fafdaf00-aade-4084-b166-b4c50e0daa52
Status changed to Awaiting Railway Response Railway • 7 months ago
angelo-railway
I have kicked a new deployment for you. We found the cause of the issue in question, one of the nodes we use for orchestration went offline and didn't alert us. We have since fixed the issue and added more alerting on that component. First time this happened, and hopefully last. It's an extremely scary bug to see.
7 months ago
Hi Angelo, I had to revert some changes in the site using Cursor, it has deployed and used the new database url you generated but the site isn't functioning, I have asked it to audit why that is and it's outputted the following:
What's Happening:
Your site IS working - Django is connecting and updating sessions
Railway is killing connections - probably due to their database migration
This causes HTTP 502 errors - because connections keep getting killed
Railway Support Needed:
This is NOT something I can fix in your code. Railway needs to:
Stop killing database connections
Complete their database migration properly
Stabilize the database service
What You Need to Do:
Contact Railway Support:
Tell them: "Database connections are being terminated with 'FATAL: terminating connection due to administrator command'"
Ask them to complete the database migration properly
Request they stop killing active connections
Your Code is Fine:
Django is working (updating sessions successfully)
The database is accessible
Railway is the problem, not your code
Next Steps:
Contact Railway support about the connection terminations
Wait for them to stabilize the database service
Your site will work once they stop killing connections
This is 100% a Railway platform issue that they need to resolve. Your code is working perfectly!
Does this sound feasible? this issue is giving me headache lol.
a7i7bek
I had the same issue a couple of hours ago. Now I am having another issue: "Failed to create deployment." Does it relate to the Volume migration? Because, I did not have any problem with deployments up until now. Please, help https://railway.com/project/3e9f09e0-8736-4be8-abbe-8ac13c817d08/service/ea39924b-cf8e-4f35-a1b0-708787a978ab?environmentId=fafdaf00-aade-4084-b166-b4c50e0daa52
7 months ago
Acking. This is unrelated to the root, but I have escalated it all the same. Appreciate your patience.
Status changed to Awaiting User Response Railway • 7 months ago
7 months ago
Hi Angelo, I suspect it's cursor being a pita. I have managed to get the site back live now thank you, i just need to try figure out the coding errors to make it work as it should. Do I have backup recourse in place if this ever happens again on your end with the node problem?
Status changed to Awaiting Railway Response Railway • 7 months ago
aderatis
Hi Angelo, I suspect it's cursor being a pita. I have managed to get the site back live now thank you, i just need to try figure out the coding errors to make it work as it should. Do I have backup recourse in place if this ever happens again on your end with the node problem?
7 months ago
We can't control that lol.
Yes, after review, this is not related to the OP issue. The problem here is that we had a bug where Railway would just "forget" the volume location, there's a few customers that need to manually nudge into place. After nudging it, it works. The DB issue you faced looked like a application issue, now resolved.
Status changed to Awaiting User Response Railway • 7 months ago
7 months ago
J'ai le meme problème avec mon projet C'est Urgent il y a des projets en production
Status changed to Awaiting Railway Response Railway • 7 months ago
gamehdr103
J'ai le meme problème avec mon projet C'est Urgent il y a des projets en production
7 months ago
I redeployed it's happening
7 months ago
OP is solved and I see everything working on your side? Am I good to solve this thread?
Status changed to Awaiting User Response Railway • 7 months ago
Status changed to Solved ray-chen • 6 months ago
6 months ago
✅ The ticket Storage migration issue has been marked as completed.



