Attempting to switch off of metal to see if that helps

look at error logs

What should I be looking at?

https://railway.com/project/28f95e16-ecc7-4cb3-82e6-7b22d4f9eefe/service/b2e9ad49-c457-4b4f-81b0-7c1cbe04d6a4/metrics?environmentId=5855d21e-0aa3-4962-b7d2-8a6cd4031189

Please see Metrics, you'll see the 503 influx

More info: Same machine same service in the most recent hop on cache-iad-kjyo7100048-IAD

Have moved regions and redeployed

Still nothing

Is anyone looking into this?

503 is returned from your app though?

It is proxying calls to a downstream

and soemthing is happening between railway and fastly

No commit / deployment changes in either application around time of error

Same machine same service in the most recent hop on cache-iad-kjyo7100048-IAD: scJ4iSV4UkyHDI2juJNzrcUzYk/oO1PMCyYg4bz5Utg!CMH!cache-cmh1290116-CMH, scJ4iSV4UkyHDI2juJNzrcUzYk/oO1PMCyYg4bz5Utg!IAD!cache-iad-kjyo7100048-IAD

cache-iad-kjyo7100048-IAD and cache-cmh1290116-CMH -- these are Fastly edge cache node identifiers. IAD = Ashburn, Virginia datacenter. CMH = Columbus, Ohio datacenter. Fastly uses airport codes for their PoPs (Points of Presence).
"Same machine same service in the most recent hop" -- this is a specific Fastly error message. It means a request arrived at a cache node that already handled it, indicating a routing loop.
Railway uses Fastly as its CDN/proxy layer for *.up.railway.app domains. So when your app hits APPURL, the request goes through Fastly first.
503 status + that specific error string is Fastly's way of saying "I can't reach the origin server" or "the request is looping." It's not your app or your backend code returning 503 -- it's the infrastructure layer in front of it.

I can get on a call if needed as well

I have a similar setup and am seeing 503s as well.

I appreciate you.

Going to try swapping to a custom domain to see if it fixes the issue on railways side

Attempting a production fix by using custom domains as I really do believe this is an issue on Railway's side

Fix did not work

Made a separate report: https://discord.com/channels/713503345364697088/1476722007789015220

Timestamp of errors starting

hi

domain?

wait

Okay

Tried something

lets see if errors go down

Can I get a redeploy rq?

yes please

on it

We had to re-roll Fastly because of… another… DDoS, that lasted 45 seconds

this may or may not be related

Didn't work

Did redeploy

Likely not network then

looking deeper

thanks let me know what you need from me and i'll be here

Wait

Going to move everything off Fastly

I have already tried custom domains incase it was a proxying issue at railway.app

thanks dude

okay, give it a sec

ok, will I need to redeploy

I don't think so

okay, everything off Fastly on your side

I attempted redeploy

if still not, will jump on bridge

Still seeing 503s

okay, lemme get a meet spun up

Logs here https://railway.com/project/28f95e16-ecc7-4cb3-82e6-7b22d4f9eefe/service/b2e9ad49-c457-4b4f-81b0-7c1cbe04d6a4?environmentId=5855d21e-0aa3-4962-b7d2-8a6cd4031189&id=668ce3d1-d7f7-429c-8c22-210699e4c63a#deploy

request sent

accepted

Was good for a bit, happening again now

Is this a specific endpoint?

we got it solved, put em off fastly

sorry, needed to exfil that