247e5ad5-c8c4-41c3-9205-3014cc65df86

Based in florida - so we're good: https://rlwy-edge-info-production.up.railway.app/

edge=railway/us-east4-eqdc4a
zone=us-east4-eqdc4a
ip=::ffff:100.64.0.4
forwarded=129.222.201.212
hs=5cf4fc755882
req=W7udzeZKSuu_-klkpKirug_28081791

Hello!

So yes all your staging services are on Metal, but your domain still uses the GCP edge, would you please enable the Metal edge for your staging domain and let me know if that helps any?

How do you do that ?

The little thunderbolt

Oh gosh you guys need to put a warning or something. This is not obvious

Its a DNS update so please wait several minutes before giving your final verdict

its more like a beta feature right now

I live on the eastern side of the continent too, would you be able to provide two endpoints, one from staging and one from prod that I could test as well?

Alright I migrated the production service (again) to METAL:

nslookup www.speleodb.org
Server:        1.0.0.1
Address:    1.0.0.1#53

Non-authoritative answer:
www.speleodb.org    canonical name = ukdjqx9p.up.railway.app.
ukdjqx9p.up.railway.app    canonical name = edge.railway.app.
Name:    edge.railway.app
Address: 66.33.22.4
Name:    edge.railway.app
Address: 66.33.22.3
Name:    edge.railway.app
Address: 66.33.22.1
Name:    edge.railway.app
Address: 66.33.22.2

It still is considerably slower. And I'm near certain it's the DB requests. Pages that are heavy on DB requests are now much slower to load.

I live on the eastern side of the continent too, would you be able to provide two endpoints, one from staging and one from prod that I could test as well?
I dont think you will see much. You need to authenticate and query some data to get some database interaction.

Can you check if there's anything incorrect between the "web frontend" and the PGSQL server ?

I'm almost certain it's database related. A page that took < 1 sec to load now takes 5+ seconds to load.

Everything looks fine to me, you are sure you are connecting to the db via the private network right?

postgresql://postgres:@postgres.railway.internal:5432/railway

Forgive me for asking, but are you positive your code is using that environment variable set in the service variables?

Absolutely certain there's no other way for my database to "guess the credentials"

Not guess, but hardcoded in some way, I've seen it lots

Nope nope.
Could it be that the "internal URL" goes first to the old datacenter and then back to US East ?

I can't see that possible, the IPv6 address has the target container encoded directly into it

Can you move the release command into a pre-deploy command and then disable the V2 builder

I'm not sure if that's normal. But the traffic pattern of the DB service is very different. It used to never go flat. And there used to not be any yellow.
Is "outbound" outside of datacenter or outside of VM/container/server ?

Because if it was to mean that the traffic is going "outside the datacenter" or outside some perimeter that would explain the problem

outbound is any traffic that leaves the private network

but lets eliminate all variables first, can you make the chagnes I suggested?

I cant - no build are functioning at all

Seems like the issue was limited to my staging environment. Let's see

Yep that exact error is why I want you (everyone) off the v2 builder

Done. And still very slow

should i be looking at the prod service? its still using the v2 builder.

Well I cant deactivate it

On any service. I click on it and it reappears

mind if i try?

Sure

maybe it's my railway.toml

# Documentation: https://docs.railway.com/reference/config-as-code#nixpacks-version
[build]
nixpacksPlan = { "providers" = ["python"] }
builder = "NIXPACKS"
nixpacksVersion = "1.30.0"                  # https://github.com/railwayapp/nixpacks/releases/
buildEnvironment = "V2"
nixpacksConfigPath = "./nixpacks.toml"

[deploy]
runtime = "V2"
numReplicas = 1
sleepApplication = false
restartPolicyType = "ON_FAILURE"
restartPolicyMaxRetries = 10

[variables]
NIXPACKS_PYTHON_VERSION = "3.13" # https://nixpacks.com/docs/providers/python

yeah that would do it

What parameters need to go ?

you are also running post_compile twice?

just remove the line

Doesnt really matter. But I removed the unnecessary one.

Which line ?

buildEnvironment = "V2"

Alright. Building

Looks like it broke the release … Now django panicks because it complains some artifacts are missing.

Alright I found a workaround for tonight.
Looks like disabling the V2 builder did indeed reduce the latency. I think it's still a little slower but not 5x anymore

do you have any tracing in your app, otherwise we are just guessing here

Came here because my application also running slow, turned out it was because of this issue

May I ask what issue, I think we have covered a few topics in this thread.

The metal edge issue, I have enabled it but it didn't seems to help

Alright, let's go back to your thread