Problem with Metal migration
kuhamaven
PROOP

a year ago

Just a small heads up. 14 days ago I changed my service and db to run on Metal, and I had no issues.

Today, suddenly, lots of users reported not being able to log in. As soon as I check the railway site I see this picture.

Thing is, it "Automatically switched me to metal" even though I was already in metal for two weeks, but not only that, it deployed a version that was a week old, which can be seen in the other picture.

I don't know why this happened, or why did it "auto switch" despite already being on metal, but the deploying a very old commit broke all the clients data as there had been a big update between those versions.

19 Replies

kuhamaven
PROOP

a year ago

c17693fa-8f81-4f00-ad4d-8f231f687806


a year ago

Likely reason for the redeploy is the recent incident in <#846875565357006878>
it deploying a 2 week old version is definitely funny though, it's supposed to redeploy the most recent active version


kuhamaven
PROOP

a year ago

It was absolutely funny getting hundreds of complaints from affected users.


a year ago

oh sorry I didn't mean to make fun your situation, I meant funny as'n weird, not funny as'n humorous


kuhamaven
PROOP

a year ago

Yeah, totally get it! NGL, It would be hilarious in many other contexts haha


kuhamaven
PROOP

a year ago

But the situation itself made it heavy on the team


a year ago

yea very sorry about that


a year ago

Just to clarify, your last active deploy wasn't that old commit, right?


a year ago

Just want to make sure before I bring this up to the team


a year ago

Also would you be able to provide a service id?


kuhamaven
PROOP

a year ago

Nope, it wasn't


kuhamaven
PROOP

a year ago

1360001788509552600


kuhamaven
PROOP

a year ago

1 was the last working deploy I had done
2 was the week old commit that got deployed by this migration that happened on railway's end
3 was the redeploy that happened when I saw it failing and was like "Weird, let's try the good ol' reset before freaking out"
4 was when I realized it was a week old commit suddenly, and re deployed the latest one.


kuhamaven
PROOP

a year ago

9455b1c0-0316-4c9f-bc1b-9d412d7bb851


a year ago

thanks for the explanation


a year ago

I'll share this with the team


kuhamaven
PROOP

a year ago

any news on this?


a year ago

no news, sorry
the team is on break though so delay is to be expected


a year ago

and they don't work on weekends anyway


Loading...