High network latency between app container and supabase database.
sergiotapia
PROOP

2 months ago

Project ID: ac62f702-07b5-4319-85e4-aab729599d4a

Did anything happen recently in how Railway (US West California) might network connect to Supabase? Network wise I mean.

Simple SQL Ecto queries would take 5ms each (great) and suddenly with no code changes these same queries started taking 50ms. Seems to have started December 22.

Did some more troubleshooting. The queries themselves execute in 0.361ms (confirmed via EXPLAIN ANALYZE), so it's purely a network latency issue.

Setup:
Railway: US West (California, USA)
Supabase: West US (North California), AWS m6g.large

Testing from Railway container (railway run):
TCP connect to Supabase: 77ms
Full psql SELECT 1: ~650ms (includes SSL handshake + auth)
In-app queries (pooled connections): ~50ms each (per AppSignal)

This was previously working fast with queries taking ~5ms total. The 77ms TCP latency suggests the traffic isn't actually routing same-region. Has something changed with Railway's US West networking or egress routing?

26 Replies

sergiotapia
PROOP

2 months ago

Also happy to move this project to Metal if that would solve the issue but I only see these options in the Region dropdown:

1453775721213329700


2 months ago

nothing has changed on our side, can you link the project and send the database url you're using?


sergiotapia
PROOP

2 months ago

Yes the project ID is: Project ID: ac62f702-07b5-4319-85e4-aab729599d4a


2 months ago

I can see your deployment was bursted to Google Cloud due to capacity constraints, and the latency from the GCP us-west1 region is much worse than metal (which you were on before)


sergiotapia
PROOP

2 months ago

What does this mean, "capacity constraints"? like my code was slow or railway had too many people?


2 months ago

Latency from GCP to Supabase is 7ms and from metal its 1ms


2 months ago

The latter, too many people were deploying at once so our orchestrator decided to offload capacity onto cloud compute


sergiotapia
PROOP

2 months ago

I see would it be possible to stay on railway? I've upgraded us to Pro a few minutes ago


2 months ago

Yeah of course, and that's the default usually. Railway's growth has just been extreme so we've had to rely more on cloud compute while our hardware orders are on the way


2 months ago

I can pin your deployment to one of our metal host machines


2 months ago

which service do you want pinned?


sergiotapia
PROOP

2 months ago

thank you so much! please let me know once it's live or what I need to do and i'll keep an eye on things

The project is callie-backend


sergiotapia
PROOP

2 months ago

in the Callie -> production env


2 months ago

ok, done!


sergiotapia
PROOP

2 months ago

appreciate it merry christmas!


2 months ago

merry christmas <:salute:1137099685417451530>


sergiotapia
PROOP

2 months ago

hm very strange pushed a simple change to my code, and it's been stuck on this step for a long time.

1453799308016091100


sergiotapia
PROOP

2 months ago

Deploy logs are also completely empty

1453799427683782700


sergiotapia
PROOP

2 months ago

it finally timed out i guess? still no logs

1453803700874379300


2 months ago

hmm one sec


2 months ago

could you go here, hit cmd+k (or ctrl+k) and search for "Deploy latest commit", then hit enter?


sergiotapia
PROOP

2 months ago

Done, it's restarting the deploy now


sergiotapia
PROOP

2 months ago

it went through very quickly it's live now, interesting


2 months ago

yeah my bad, I accidentally pinned it to an invalid host


2 months ago

you should be good now 👍


sergiotapia
PROOP

2 months ago

thank you!


Loading...