Hey there! We've found the following might help you get unblocked faster:

• 🧵 High network latency between app container and supabase database.

• 🧵 Long(ish) network latency between Railway and Planetscale

• 📚 Railway Metal

If you find the answer from one of these, please let us know by solving the thread!

Howdy!
Would love to clarify, are you using internal networking on it?

Yes, we are using internal networking from web server to redis server.

Hello,
The current behavior is expected; we distribute workloads to either zone depending on availability at any given time, and we don't currently offer the ability to pin workloads to a specific zone.
Best,
Brody

It's not acceptable for my cache reads to take 40x longer based on random zone distribution. Is this something that can be addressed soon or do I need to keep redeploying several times until replicas are on the same zone for the foreseeable future? Do all other regions have the same randomness?

All regions have availability zones, but this split is most common in us-west, given the distance between the two zones.

Can you answer the first question as well?

We have no defined timeline for changing this behavior.

To be explicit -- the best (and only) way to ensure that replicas are on the same zone as the cache server is to manually check post-deployment and redeploy repeatedly until they are are on the same zone? This seems extremely inefficient so I want to triple-check that this is the official recommendation to accomplish this goal and have 40x faster cache reads.

Update: we just lost a whole day of development because apparently this issue also occurs when there is only a single instance and not just with replicas. This issue caused our staging environment to get thousands of timeouts for hours until we realized it's the same issue.

It's hard to debug as the Deployments tab shows the web app and redis server are both on "us-west2" but when you look at the Metrics tab, and then click on Replicas (vs Sum), you can see the real zone where redis is on us-west2 and the web app is on us-west1.

Since staging auto-deploys, we need a way to programmatically fetch the real zone so we can compare and initiate a redeploy. Please advise on how to do this.

Hi,

Sorry for your struggles. At this point we do not expose this information programatically.

However, if you are willing to run a single replica service we could pin it to a the Metal us-west2 region.

Let us know if you'd like to do that.

Kind regards,

Sam

Appreciate the offer -- we would definitely like to do this for our staging environment with auto-deploys to ensure consistent behavior. I think this thread is public so what's the best way to share that information with you? You could probably find it pretty easily too -- Staging Env + Rails app name that ends in "-backend".

For production, we ran into performance issues when only running a single replica so I'd prefer to keep 2 instances there. If you have an internal feature request process, please add one for me to be able to configure this via railway.json.

Thanks for your note. I think I see the environment you are talking about and will follow up.

We are working on resolving the underlying issue as well.