03366656-1bad-4591-bfdd-134b17afd697

@Nico - data loss via migration?

point us to the service in question please

"Postgres 2.0" (https://railway.app/project/03366656-1bad-4591-bfdd-134b17afd697/service/1a102620-c9d6-446d-8a00-ed0ed496ba70), volume "pgdata" (https://railway.app/project/03366656-1bad-4591-bfdd-134b17afd697/volume/cbbc2db4-611b-41b3-9f92-5e9a27962265/metrics)

thank you

ack, looking into this

I had previously migrated 2 other volumes (for CZ3 OTA and CZ3 Assets) and they both went smoothly.

I've got a dump of basically all important data from earlier today, so if the data from just prior to migration can't be restored, I should be able to restore that dump if necessary --- but if I'm able to get the data from just prior to migration restored, that would be better.

Just to confirm, is the data actually missing, or could it be the usage metric only?

Gone so far as Postgres suggesting database does not exist.

thanks, we are investigating

Thanks for the quick responses!

Was feeling pretty hopeful about this working correctly based on the 2 larger, unimportant ones migrating correctly - wish one of those had failed instead of this! 😂

really sorry about this

the migration workflow appears to have worked correctly (taking a snapshot and restoring it to the new volume)

we are investigating why this led to an empty volume

we'll keep you posted as soon as we have an update

👍

One thing I realise I didn't mention initially, but I realise now might be important - at the same time as switching the service's region, I also changed the runtime from Legacy to V2.

Not sure if that's relevant, but thanks for the info

Still looking into the issue, will keep you posted

Do you think there's any chance of getting the data from prior to the migration restored?

If this isn't likely, I'd probably prefer to sooner-than-later spend some time restore my older backup and get caches refilling, and deal with the slight data loss.

I just recovered a dump from 3:41 (UTC+2)

I can send it over. Are you ok with restoring a dump yourself?

Depending on which dump is the freshest (yours or the one I just recovered), I can send it over

Looks like my dump is slightly fresher -- but yours might be more complete, so would appreciate having that just in case there's any data I need from it that isn't in mine.

(just to be clear 3:41pm UTC+2)

I'll send it over

Thanks!

The dump is too big for discord, I'm posting it on gcs, will be right back

I'll send you the link via dm

could you please accept the request?

Accepted

Once again, apologies for this issue

We opened a ticket to investigate how a seemingly successful migration ended up with an empty volume

Hey @sohcah, I've gone ahead and added $250 of credits to your team for the trouble caused by the issue. Hope this helps !

Thank you, that is most appreciated!

Appreciate all of the quick responses and support here, especially on a US Public Holiday!

hey @sohcah just wanted to circle back, where you able to restore the dump and resume operations?

Yep - got all public systems back up and running yesterday.

Still a few bits for some internal systems I need to clean up, but I've got all of the data I need to do so, and that's very low priority.

awesome, glad you where able to get everything back up and running, do you think it's okay to mark this as solved?