1b0d51d3-f173-4fd6-bf29-f0025eae2d1a

I think this is a service specific issue - my services are showing 100-200ms on public domain

Are you putting your service in a distant region (e.g. us -> eu)?

Yes. I’m in EU, all services are in US. I’ll check this direction once Amsterdam becomes available. Just don’t want to waste time if it’s something with Hobby. 1000ms is on the higher end and as stated the server itself is returning pretty fast.

Do you have traces enabled to back that up?

It's not a hobby specific issue, railway doesn't change networking based on your plan

I have the WSGI server logs which measure the time from the time the request is received and until it's fully processed.

And a lot more. Pretty sure most of the time is happening outside the WSGI server.

     time_namelookup:  0.112218s
        time_connect:  0.189007s
     time_appconnect:  0.343297s
    time_pretransfer:  0.343528s
       time_redirect:  0.000000s
  time_starttransfer:  0.750120s
                     ----------
          time_total:  0.750215s

100.64.0.6 - - [06/Feb/2025:12:58:18 +0000] 'GET /REDACTED HTTP/1.1' 302 0 '-' 'curl/8.7.1' in 24472µs

I'm willing to deploy a route that simply returns a 200 without any processing if that will help looking into this.

Even health check route that’s returning 3000 microseconds is taking 700ms to reach. This is too much for geography to account for.

The last time someone in the help thread was having latency issues was related to not being on runtime v2

Just out of curiosity, is this service of yours on the runtime v2?

was it created after 2024/06/04 (YYYY/MM/DD)?

Yeah, I saw that thread while researching. The service is less than a week old. And I verified that it’s on V2.

I’m gonna try to move it to us-east-1 (alongside DB and Redis) and see if that makes any difference.

Oh, I can’t move Postgres to us-east-1.

Quite interesting. Even with the DB in another region, I’m seeing sub 500ms responses for the first time. Still far too slow.

Oh, I can’t in the Hobby plan.

make sure your app and db are in the same region!

same zone too

They are

The response times I’ve listed include the trips to the DB.

Then yeah just looks like it's geographical distance coming into play now

That’s interesting

I think it’s still too much to account for that

got a command you'd like me to run? I'm closer to US-West than you are

EDITED OUT

what do you use for these printouts?

the first one's source is based on: https://stackoverflow.com/a/22625150/14207055

the 2nd is server logs from Railway (which I can access)

yay windows option too

oh Brody, I'm so sorry for you

and here is a request to a service on us-east metal (i am much more east) -

\>curltime.bat https://utilities-metal.up.railway.app/ok200
     time_namelookup:  0.048874s
        time_connect:  0.100423s
     time_appconnect:  0.155786s
    time_pretransfer:  0.155836s
       time_redirect:  0.000000s
  time_starttransfer:  0.270153s
                     ----------
          time_total:  0.270172s

that's more like the number I'd like to see (270ms) since this route is doing absolutely nothing.

500ms for that seem a bit high for me, especially since for [SOME COMPETITOR], even a US route is returning at sub 250ms consistently (and that route is doing something).

270ms*

yeah yeah, sorry

fixed that typo

so yeah this is just a distance thing

I'm doubtful it's all due to geolocation. I'm sure some of it is, 500ms seems much.

Again, I'm seeing servers in California returning a response at sub 250ms consistently for another provider, and that's for a route I know has to do some lookups.

At first I thought "yeah they're much better engineers / using better mechanisms / etc...", but then I saw my server response times, and even the route I gave you at /up does literally nothing.

I guess I'll wait for Amsterdam to see and evaluate my options.

did you want me to run the utilities in Amsterdam?

no no, what I'm saying is that Railway US west vs a competitor hosted elsewhere US west is a no contest.

My route that does nothing takes >700ms from where I am, theirs takes less than 280ms 99% of the time.

Something may be missing in my tests, but I'll have to evaluate.

That's why I'd like to wait to Amsterdam, because I'm much closer to it, in order to weigh on how much latency Railway's LBs or whatever add into the mix.

yeah and thats why i offered to run the utilities in Amsterdam

sorry, I don't get the point of that (meaning, I literally don't understand the purpose of that test)

can you elaborate?

you want to deploy something to Amsterdam.

you cannot deploy something to Amsterdam since you are hobby.

i have offered to deploy something to Amsterdam for you.

Like, my app? Or the test app you’re using? Either way sounds interesting

my test app, you mentioned you just want something that returns 200

Sure. Thank you.

will do

done -

gcp since eu-west metal isnt ready for workloads yet

Between 0.25-0.5 seconds

Same app deployed in US east in another provider:

     time_namelookup:  0.002456s
        time_connect:  0.016633s
     time_appconnect:  0.146623s
    time_pretransfer:  0.146761s
       time_redirect:  0.000000s
  time_starttransfer:  0.328444s
                     ----------
          time_total:  0.328547s

Would be awesome if Railway engineers were at least willing to conisder this.

consider what?

consider that Railway adds significant latency

that cannot be accounted for due to geography, or application code

theres a lot more that goes into play that railway cannot control

I'm sure but when it's +2x than the competitors, I'd at least want to rule that out. But, it's understandable.

try us east metal -

i guess it wont make a difference the edge proxies are still gcp

I can't move the DB to US east (yet).

I guess I'll give Pro a spin soon enough to make a fair assessment.

let me know how that works!

With Pro, i'll be able to move the DB off west and to east right?

yep, and even to EU

us-east-1 is much better than us-west-1

around 0.45-0.50s most times, still slower than the competition

Trying Amsterdam

in my humble opinion, you should wait until we have the edge proxies on metal too, then make your final decision, right now, no matter if you use gcp or metal, you are still talking to gcp first

in my humble opinion, you should wait until we have the edge proxies on metal too

Is there an ETA?

I very much like Railway, and have some time to decide.

been banging my head trying to deploy properly on that other platform

So Brody do you mean that right now on Metal, even though the machines are better, you're still "behind" gcp?

Because I just moved to Amsterdam, and this is pretty sweet:

     time_namelookup:  0.002964s
        time_connect:  0.080993s
     time_appconnect:  0.157722s
    time_pretransfer:  0.158244s
       time_redirect:  0.000000s
  time_starttransfer:  0.231267s

Even though it's not metal.

yes your http request still hits our edge proxy on gcp first no matter what, eta for proxy on metal should be by the end of this Q

so if I understand correctly, non-metal already have edge-proxies?

i mean metal or non-metal, any http request will still hit the edge proxies that are gcp only right now

oh, got it

metal edge proxies coming by the end of this Q, hopefully at least.

our metal data centers have better routing and better isp hookups, so the time should improve, at least thats the goal

OK, tested the same exact route.

Railway in Amsterdam (DB & Redis in the same region), 4,500 km:

0.706260s

0.579180s

0.372202s

Competitor in Frankfurt, 4,000 km:

0.287137s

0.290455s

0.288477s

us-east-1 vs. us-east-1: competition (from my location) just blows Railway away.

Railway's deployment flow and UI are light years more convenient to me, but this service is going to be a bit sensitive in terms of timing.

I'm definitely looking forward to edge proxies and hopefully don't have to make the decision before then. Reverting to Hobby until then and will test again.

drop the competitor's name, this is a safe space

Vercel.

I didn't want to come off as trying to spam/advertise. Thank you.

ah yea AWS, they have far better routing than GCP

hopefully metal will have better than both, but only time will tell

Yeah. In my day job I work extensively with all 3 cloud providers directly and have been doing so for >4 years. My guess of why Railway picked GCP is startup credits (totally legit) because working with the product (GCP) sometimes feels hostile. Hopefully metal routing will be comparable.

from what ive heard, we picked gcp because its more put together than aws, more well rounded, seems like a cohesive product, could be wrong, who knows, long pre-dates my time with railway

Let’s see what happens with the in-house proxies. One thing is sure: I wouldn’t even have stayed with Railway if not for your help (in ~Jan 2024 you helped me put together my first project on here)

ouuuu tell me about this project

It sucked and failed, like most of them.

yeah but I wanna know what it is, I forgor these things

@foo229 - we would be honored if you can give it a try - https://discord.com/channels/713503345364697088/1344913382427332670