How to turn on edge-proxy using config as code or via public api?

Could you look into some of these reports that the edge proxy is not 100% compatible with Safari?

https://help.railway.app/questions/django-safari-pwa-7246187c

Our new edge proxy has been powering >10% of all Railway traffic since launch. For the past two months, we have fixed all critical bugs related to reliability, and have been improving its stability across the board.

We are now ready to begin moving more traffic to the new edge proxy.

Starting next week, we will begin moving all Railway-generated domains (*.up.railway.app) to our new edge proxy. Custom domains will remain on our legacy proxy unless you have opt-ed into the new proxy via the “Enable New Proxy” toggle in your service’s settings.

This change will be entirely transparent to users with zero expected downtime. No action is required on your part. If you run into network-related issues without any code changes or recent deployments, please reach out to us by leaving a reply here.

We just ran into a strange issue and it turned out to be likely related to one of our deployments being routed through the new edge proxy now.

We have this "proxy endpoint" on our frontend app that takes requests from our frontend, injects an access token, and forwards the request back to the client: https://github.com/drips-network/app/blob/main/src/routes/api/multiplayer/%5B...path%5D/%2Bserver.ts

Today, suddenly, without any changes from our end, the staging deployment of our app broke because it wasn't able to parse responses from this proxy endpoint anymore. Seemingly nothing had changed, and direct requests to the express app also still worked just fine, so it was a bit of a mystery. Meanwhile, the prod app (1:1 same as staging except it's connected to the staging instance of the express server) still worked fine.

Turns out that the staging instance was being routed through the edge proxy now. And apparently that edge proxy is returning gzipped responses when it wasn't previously (?). And because we were just forwarding the raw Response from the express server, it was forwarding the new Content-Encoding header seemingly added by the Edge proxy too, which caused decoding to fail client-side because it actually got a non-compressed payload.

I ended up fixing this by just constructing a new Request object server-side instead of returning the original response from the express server directly: https://github.com/drips-network/app/pull/1170/files

Not complaining because we should've done that from day 1, just though I'd post here in case anyone else falls into this pit :) Glad it affected our staging server before prod.

This likely happened because all new domains are on the v2 HTTP Proxy. If a PR created a new environment in staging than it would have made a new domain and thus use the new proxy.

Out of curiosity, could you perhaps share some engineering details on how you built this proxy from scratch and made sure it's reliable and performant enough to replace your previous solution?

It would be interesting to know which language platform you chose (Rust? Go?) and if you found the ecosystem of that platform suitable for building this proxy (I assume you didn't implement HTTP from scratch ).

Our new edge proxy has been powering >10% of all Railway traffic since launch. For the past two months, we have fixed all critical bugs related to reliability, and have been improving its stability across the board.

We are now ready to begin moving more traffic to the new edge proxy.

Starting next week, we will begin moving all Railway-generated domains (*.up.railway.app) to our new edge proxy. Custom domains will remain on our legacy proxy unless you have opt-ed into the new proxy via the “Enable New Proxy” toggle in your service’s settings.

This change will be entirely transparent to users with zero expected downtime. No action is required on your part. If you run into network-related issues without any code changes or recent deployments, please reach out to us by leaving a reply here.

well this aged like fine wine

You just *** my buisness for 1 hour, thanks a lot guys

You just *** my buisness for 1 hour, thanks a lot guys

This incident made me realize how important it is to always have a backup... Going to work on making one now. Railway you should do the same.

Sorry, I can't stand griping like this. Nobody wants an outage like this but it will happen on any service. If your business is – whatever three asterisks means – by an outage then it behooves you to have some kind of failover strategy even if that means it is on a different service. Don't complain, get redundant!

Deepest apologies to anybody affected by this.

Unfortunately, this was a process error which had nothing to do with the technical side of the proxy. It was an issue with our Terraform stack which cycled the proxies simultaneously.

We had 2 different fallbacks fail here. We've put in additional fallback measures to guarantee that this is not an issue in the future

You can read the full post mortem below.

https://blog.railway.app/p/incident-august-27-2024

Thank you,

Jake Cooper

Founder @ Railway

You just *** my buisness for 1 hour, thanks a lot guys

Deepest apologies here. I'm going to reach out personally over email right now.

Going to unpin this thread since we have moved over everyone to the new proxy with final issues resolved. Thank you everyone who's helped us build a more stable platform.