22 days ago
Hello - after auto-migration to metal it looks like my postgres database is no longer showing up under the data tab or when i try and connect to it through the API. I'm hoping maybe it's just an issue with it mounting. Please advice.
5 Replies
22 days ago
Hi Grant. Looking into this. It looks like a few other instances were migrated as well for you
prj=css-paint / svc=Postgres / env=production / deployId=fae174
prj=collage / svc=Postgres / env=production / deployId=660d76
prj=quotes / svc=Postgres / env=production / deployId=17c45f
prj=cascade / svc=Postgres / env=production / deployId=3e35e9
prj=db-feed / svc=Postgres / env=production / deployId=7138da
Do any of those have this issue? It looks like the migration completed successfully, and then, shortly after it started, there was a drop in the data
Attachments
Status changed to Awaiting User Response railway[bot] • 22 days ago
22 days ago
I'm looking into this with the team but, just wanted to see what the DB is used for an if there's a possibility some kinda of external job fired (like, what's upstream of this Postgres?)
22 days ago
Figured it out. Wow that's a doozie
So, when we create the migration, we don't actually move the old instance. We make a copy, streaming replicated, verify integrity, and then cut over.
It appears your volume was migrated during a high traffic time, which meant the migration timed out. Which, would have been fine, except it was retried. And when it retried, it tried to copy from new (empty) volume. We performed this migration 10-20k times yesterday, and it looks like < 10 instances right now were affected.
We've done the following:
- Fixed the bug
- Reset the migration, restoring your data
- Applied credits to your account for the inconvenience
Again, very sorry this happened. I can assure you the data is there, and that we take your trust in us, and your data, VERY seriously.
Best,
Jake Cooper
Founder @ Railway
22 days ago
So glad it's back! Thanks for figuring it out and the explanation.
Status changed to Awaiting Railway Response railway[bot] • 22 days ago
21 days ago
Thanks for sticking with us through the migration!
Status changed to Awaiting User Response railway[bot] • 21 days ago
Status changed to Solved echohack • 21 days ago