Problem with Metal migration
kuhamaven
PROOP

10 months ago

Just a small heads up. 14 days ago I changed my service and db to run on Metal, and I had no issues.

Today, suddenly, lots of users reported not being able to log in. As soon as I check the railway site I see this picture.

Thing is, it "Automatically switched me to metal" even though I was already in metal for two weeks, but not only that, it deployed a version that was a week old, which can be seen in the other picture.

I don't know why this happened, or why did it "auto switch" despite already being on metal, but the deploying a very old commit broke all the clients data as there had been a big update between those versions.

19 Replies

kuhamaven
PROOP

10 months ago

c17693fa-8f81-4f00-ad4d-8f231f687806


dev
MODERATOR

10 months ago

Likely reason for the redeploy is the recent incident in <#846875565357006878>
it deploying a 2 week old version is definitely funny though, it's supposed to redeploy the most recent active version


kuhamaven
PROOP

10 months ago

It was absolutely funny getting hundreds of complaints from affected users.


dev
MODERATOR

10 months ago

oh sorry I didn't mean to make fun your situation, I meant funny as'n weird, not funny as'n humorous


kuhamaven
PROOP

10 months ago

Yeah, totally get it! NGL, It would be hilarious in many other contexts haha


kuhamaven
PROOP

10 months ago

But the situation itself made it heavy on the team


dev
MODERATOR

10 months ago

yea very sorry about that


dev
MODERATOR

10 months ago

Just to clarify, your last active deploy wasn't that old commit, right?


dev
MODERATOR

10 months ago

Just want to make sure before I bring this up to the team


dev
MODERATOR

10 months ago

Also would you be able to provide a service id?


kuhamaven
PROOP

10 months ago

Nope, it wasn't


kuhamaven
PROOP

10 months ago

1360001788509552600


kuhamaven
PROOP

10 months ago

1 was the last working deploy I had done
2 was the week old commit that got deployed by this migration that happened on railway's end
3 was the redeploy that happened when I saw it failing and was like "Weird, let's try the good ol' reset before freaking out"
4 was when I realized it was a week old commit suddenly, and re deployed the latest one.


kuhamaven
PROOP

10 months ago

9455b1c0-0316-4c9f-bc1b-9d412d7bb851


dev
MODERATOR

10 months ago

thanks for the explanation


dev
MODERATOR

10 months ago

I'll share this with the team


kuhamaven
PROOP

10 months ago

any news on this?


dev
MODERATOR

10 months ago

no news, sorry
the team is on break though so delay is to be expected


dev
MODERATOR

10 months ago

and they don't work on weekends anyway


Loading...