might've been resolved on its own?

back green but still...

connection err closed something something when I tried to fetch my service's healthcheck endpoint

3 services were/are effected so far, no new deployments

will look around

I just had it happen on railway.com as well

I'm in eu ams

affected services whose healthchecks failed externally

Seeing latency spikes

both screenshots are of services linked above

not seeing the socket hang up anymore but still seeing slight latency spikes

socket hang up again

may I ask which ip/isp you're getting these from? can dm too

502 bad gateways, connection reset by peer

69.46.46.14:443: i/o timeout

constantly, like right now?

upstream error

uhh not super constantly but quite consistent

is there any response body?

upstream error

everything seems to be stable the last 30m, but might just be luck

let me know if you see one in the next 5min or so

instability is back...

unexpected EOF

read tcp 172.18.0.8:33922->66.33.22.216:443: read: connection reset by peer

ok, I know why you see this now

wait what

Oh I see, the origin/railway proxies (on the 66.33.22.0/24 prefix) were just deployed

How aggressively are you monitoring those? Do you keep the connection open?

I don't think so, minutely checks

I'm trying to access the failing URLs from my browser, I get hit with a conn refused (or alike, browser autorefreshes) and afterwards it seems to work fine

could you send me the failing URL in here/over DMs? (does it resolve to an address within 66.33.22.0/24)?

I'll send you the ones my monitors are annoying me about, do want to note that it's intermittent so far

seeing timeouts, question mark

do you know if its the 66.33.22 IPs or 69.46.46 IPs you're seeing timeouts on?

69.46.46.58

resolved ""itself"" as of now but

ok, that would do it. bgp should reconverge

just got 5 minutes timeout then it fixed itself

Let me know if you see anything else.

Was this from a monitor out of interest? Is the monitor hitting the "ams1" POP? https:///.railway/cdn-trace

Yeah, it was a monitor hitting my own API. The entire app was running into a Cloudflare timeout error, but the Railway metrics were still showing everything as healthy and operating normally so idk what was that

https://discord.com/channels/713503345364697088/1511663784127762483/1511663784127762483 seeing some latency spikes

69.46.46.58

Could you run a traceroute/mtr to that IP from the monitor's network/server if possible? 🙏

not consistent latency... mtr just shows ams eq6 right now, but ill keep retrying

some cloudflare proxied requests are timing out completely

I'd like to see the hops you take before eq6

https://utilities-us-east.up.railway.app/raw

What is the x-railway-edge header you see here?

railway/europe-west4-drams3a

What about here? https://utilities-us-east-cf-proxied.railway.com/raw

same edge val, cf-ray ending in -ams

Ty

timing out for me

and... not anymore

No timeouts on https://utilities-us-east.up.railway.app/raw right - just via Cloudflare?

timeouts only observed via cloudflare so far

latency spikes on non-cloudflare still a thing but my monitors havent tripped on them

cf-ray became lhr, then timed out the following refresh

(and once again I am able to fetch it fine..)

I'm assuming you disabled the ams pop

Nope it's still up, do you see that on the non-CF domain as well?

I do not

I have a suspicion on what it could be. I'm going to disable something and we can see if that resolves it.

I've disabled something in AMS - let me know if you see improvement over the next half hour or so

still seeing these (or should I just wait a bit more)

What latency is that tracking? ICMP or HTTP?

HTTP

still going cf ams -> hikari lhr

non-proxied latency spikes still occurring

Is there a specific domain this is occuring on or is it all of your domains?

I'll dm

Monitors tripped over *.up.railway.app timeout

so it's no longer necessarily CF specific

I'm also seeing latency spikes from Singapore

(Singapore Railway service -> EU-W Railway service over pubnet)

I'm guessing something was changed about 18 minutes ago? Latency seems better

There's still few latency spikes going on, but not often

Interesting. All of those domains have pointed to the old edge/66.33.22.0/24 for about 8 hours.