Hey Harry, the auto migration is part of our infrastructure upgrade plan to migrate users to Railway Metal which already has 40K+ deployments without issues and is much faster as cheaper. Plus, as pro if 80 percent of your usage costs come from Railway Metal instances, we waive all seat costs!

You can check the service's deployment history for the exact date and time of the update but if you have any specific concerns about the migration or have any issues you're facing because of it, please let me know and I'll share it with the team and get you sorted!

Was your infrastructure upgrade plan communicated to me? The email said only hobby and free plans.

Fyi: one of the critical API calls started failing on our end and caused a massive disruption, because in your new infra there's a bug that when the payload size is big the payload upstream fails.

This is messed up.

and if communicated to me, how was it communicated? was it an email, because I don't see any.

Hey there Harry,

I am the person responsible for this rollout and the features related to making it so that you are exempt (the banner that you clicked).

I apologize for not emailing you, it must have slipped through the cracks when we did email some affected users when we accidentally moved Pro users.

However, for us, it's a bit unclear, can you tell us the timeline of when you got moved, this way we can we can find the specific workload and if it lines up to the bug that we had.

Thanks,
Angelo

Hey angelo, it got moved sometime after march 17.

So we have an api call which takes in a relatively big json payload. And that particular endpoint was throwing 502s the whole time.
and the error was something to do with: payload upstream failed http2 <some stuff>proto<some stuff> edge </stuff>

(Also, for the new metal stuff http logs take like 20 mins to show up. bad.)

This is a bit weird, we don't support Http2 even on GCP? I will need more information on the particular log. Would you mind duplicating the service so we can get the correct logs. (As for historical logs... removed deploys don't have access to old logs...)

So here's my suspicion. The workload is on Metal, the proxy... on GCP. We saw some customers who weren't on Metal edge when we moved our customers over (on accident) so the network hop looks like:

- Metal -> GCP -> Metal

That would explain the additional 502s. Still speculative, but we tend to take the customers side and then try to prove you wrong.

For now, you are exempt for any automatic move. I updated our timelines to accurate represent that we are indeed moving Trial and Hobby users over- but not Pro. We do reserve the right to move workloads around on boxes- but you should be fine for now.

Thanks,

Angelo