I would like to get some sense of what went on here for me to know how to prevent this in prod

Hey, is your database still wiped?

yes it never recovered. a new migration executed on service restart

And yes, unfortunately this was Railway's fault.

We've identified that a component in our deployment infrastructure became unresponsive, which caused your deployment to hang before eventually failing. We sincerely apologize for the frustration this caused.

We've now implemented additional monitoring to detect this type of issue immediately, so our team can resolve it much faster if it happens again.

Do you want to recover it? I can raise it to the team.

yes is that possible. its unclear to me how hang in the deployment causes a disk wipe. is there more i should be doing in production environment to prevent against this. this would have been very costly to us in production.

I'll raise this to team, unfortunately I do not know most of the answers to your question.

cc @Brody

Hello,

The issue the two other users faced would not cause any data loss.

Oh my bad then, ignore what I said.

Should I be looking at tax-postgres?

yes, same thing happened to ponder-db in same env

we did not see the hung deployment. deployments were successful but just all the data was gone

Looks like ponder-db was last deployed around 2 months ago? should I be looking at another environment? I am looking at staging right now.

just tax-postgres. ignore previous message about ponder-db

The team is looking into this now.

Would you mind checking now?

I believe it's been re-initialized correctly (happy to cover what happened after and what we'll do to fix it in a sec)

yes looks like data has been restored. reminder to encrypt at rest seeing u guys poke around in the table 😂

looking forward to full diagnostic and some advice on protecting our production services from such issues

Everything is encrypted at rest on our side!

but presumably you can view tables the same way we can through the webapp

Yes. We have a BAA for enterprise that will make it so that someone internally needs your consent

But, in this case, I actually just went off metrics

this

What happened, and I have to figure out why is that you had 2 identical volumes (as in, they had the same unique identifier)

However one of them was older. I think it might have picked an older snapshot. Why? Ill dig in

Yup

would love a full post mortem here and some advice on protecting ourselves from such events in prod. thanks

Yup. We believe this occurs when you have backups and a migration

The backup fires, the migration grabs the new partial as gospel, and then migrates that

Will get confirmation for this today

!remind me to follow up in 4 hours

(Latest)

just a heads up that he left the server

im back

this is still pretty insane to us that we had a volume just bricked. would like some further explanation and some credits

Oh, interesting. I didn't follow up

This is 100% what happened here. We patched it right after you reported it. It's definitely obscene; 1:1B+ collision racecase

Went ahead and applied some credits to your account for the issue!