Deployments issues on Metal
alex73630
PROOP

9 months ago

Project ID: 1ad2e8d9-d8df-4cd2-aca2-4be43f7219d3

Our API service started failing deployment on our production environment with a strange error.
It apparently thinks it's in Region "0" instead of the EU Metal region where the current active deployment is.
It also randomy started to ignore the config file in our repo (switched back to Nixpacks instead of Docker) and throwing errors like "Cannot create code snapshot right now" on further deployments.

I guess this service is in a broken state but as it is on our production environment, I cannot just delete and recreate it and wait for a DNS propagation to happen on the API's domain.

Solved

0 Replies

dev
MODERATOR

9 months ago

Happened to me too, very weird behaviour
I'll go ask about it


dev
MODERATOR

9 months ago

Team's working on a fix


dev
MODERATOR

9 months ago

Sorry for the trouble


dev
MODERATOR

9 months ago

Angelo from the team explained what happened here if you're curious


Hey @alex73630 -

We just got the fix in, we are going to be doing a few things.

Railway sparingly moves workloads around as a function of our ability to handle capacity. Box gets full, we move 'em. Box needs an upgrade? Move 'em.

This time, we did a type change on how we handle multi-region deployments because we need to keep track of regions across cloud providers GCP <-> Railway Metal

However, because we didn't account for that, we sent them to a new, un-working region: Region 0
Moving forward, we will:

  • Put a Railway banner whenever the platform takes an action on their behalf. Avoiding confusion if we do a restart.

  • Not send them to Region0

In your case, your existing workload is fine an operable yes?


New deploys should have gone out just fine.


alex73630
PROOP

9 months ago

Hey Angelo,
Thanks for the response.

Our current deployment is working as expected, only the newer deployments have failed to even build.
Luckily they didn't contain any new backend code and so it wasn't a huge issue.

I'll try to do trigger a deployment to see if it's fixed.


alex73630
PROOP

9 months ago

Can confirm, I was able to deploy a new build 👍
Thanks for the help !


Thanks for using Railway

(and reporting)

ty @Fragly for escalating it to us


alex73630
PROOP

9 months ago

My level of trust in Railway is the same as my current Metal usage, 99% 😄
Keep up the great work!

1349439918883078100


dev
MODERATOR

9 months ago

!s


Status changed to Solved dev 9 months ago


Loading...