Postgres database missing after auto-migration to metal

grantcusterHOBBY

22 days ago

Hello - after auto-migration to metal it looks like my postgres database is no longer showing up under the data tab or when i try and connect to it through the API. I'm hoping maybe it's just an issue with it mounting. Please advice.

Solved

5 Replies


Status changed to Awaiting User Response railway[bot] 22 days ago


22 days ago

I'm looking into this with the team but, just wanted to see what the DB is used for an if there's a possibility some kinda of external job fired (like, what's upstream of this Postgres?)


22 days ago

Figured it out. Wow that's a doozie

So, when we create the migration, we don't actually move the old instance. We make a copy, streaming replicated, verify integrity, and then cut over.

It appears your volume was migrated during a high traffic time, which meant the migration timed out. Which, would have been fine, except it was retried. And when it retried, it tried to copy from new (empty) volume. We performed this migration 10-20k times yesterday, and it looks like < 10 instances right now were affected.

We've done the following:

- Fixed the bug
- Reset the migration, restoring your data
- Applied credits to your account for the inconvenience

Again, very sorry this happened. I can assure you the data is there, and that we take your trust in us, and your data, VERY seriously.

Best,
Jake Cooper
Founder @ Railway


grantcusterHOBBY

22 days ago

So glad it's back! Thanks for figuring it out and the explanation.


Status changed to Awaiting Railway Response railway[bot] 22 days ago


echohackEMPLOYEE

21 days ago

Thanks for sticking with us through the migration!


Status changed to Awaiting User Response railway[bot] 21 days ago


Status changed to Solved echohack 21 days ago


Postgres database missing after auto-migration to metal - Railway Help Station