Slow to connect replica
timbo-tj
PROOP

2 years ago

I am using Uptime Kuma to detect down services. Our game server has a few replicas. And for some reason it seems like one of the replicas are 'slow' to connect to. Using Chrome developer tools I can see that it will randomly take a very long time (1s+) to connect to the replica. I am pinging the health-check endpoint which returns how long the check took. My server responds with <10-20ms, but the request does not resolve for over 1s+.

It only randomly happens. I can keep pinging the health-check end point and I get a lot of very quick responses. But every 4-5 attempts hits the 'slow' replica. (I assume).

Project ID: 4c3b4b0e-006a-407e-90c7-9c3031cd622f

Service: 2077a108-22c6-4367-951d-41afafbfd450

Health-check endpoint is {service url}/health-check

Maybe this will resolve on its own soon, I will keep an eye on it. For now I have set the timeout to 2s so we aren't being flodded with alerts.

34 Replies

brody
EMPLOYEE

2 years ago

Are you using the edge proxy?

timbo-tj
PROOP

2 years ago

whats the edge proxy

brody
EMPLOYEE

2 years ago

a new home grown proxy that will replace railways current envoy proxy, it's meant as a faster (to say the least) replacement

brody
EMPLOYEE

2 years ago

and it's a simple toggle switch away

timbo-tj
PROOP

2 years ago

i dont know anything about it

timbo-tj
PROOP

2 years ago

i can try it out?

brody
EMPLOYEE

2 years ago

absolutely

brody
EMPLOYEE

2 years ago

please do

timbo-tj
PROOP

2 years ago

this happened right after i pushed some changes and forced a new build, which i know would suggest its probably something i did.. but..! the pattern doesnt make sense. at all. <:3HC_think:712416049236017152>

timbo-tj
PROOP

2 years ago

maybe i can log out the instance id on health check to see if its the same instnace that is responding slowly

brody
EMPLOYEE

2 years ago

that would be a very good idea, do you know the env var for the replica id?

timbo-tj
PROOP

2 years ago

yeah i think so

timbo-tj
PROOP

2 years ago

any link to info about this edge proxy thing?

brody
EMPLOYEE

2 years ago

timbo-tj
PROOP

2 years ago

thanks

timbo-tj
PROOP

2 years ago

all seems to be working now, i did enable edge proxy but i also redeployed, well, everything

timbo-tj
PROOP

2 years ago

1249008725558034400

timbo-tj
PROOP

2 years ago

so yeah uh we are back to normal

timbo-tj
PROOP

2 years ago

<:3HC_Shrug:783043455382585354>

brody
EMPLOYEE

2 years ago

awsome, let me know if you see any more issues

timbo-tj
PROOP

2 years ago

seems like the new prooxy kills socket io

timbo-tj
PROOP

2 years ago

so ive turned of the edge proxy beta on our socket io service

timbo-tj
PROOP

2 years ago

i wonder if thats worth reporting..?

brody
EMPLOYEE

2 years ago

websockets should work just fine

brody
EMPLOYEE

2 years ago

are you getting any specfic error?

brody
EMPLOYEE

2 years ago

websockets do indeed work

1249012152434426000

timbo-tj
PROOP

2 years ago

try with a couple of replicas?

timbo-tj
PROOP

2 years ago

i suspect the handshake/upgrade is failing

brody
EMPLOYEE

2 years ago

are you on the legacy or v2 runtime?

timbo-tj
PROOP

2 years ago

V2

brody
EMPLOYEE

2 years ago

i cant reproduce any websocket errors with the v2 or the legacy runtime, or the edge proxy, please share your errors

timbo-tj
PROOP

2 years ago

I'll investigate and open a new thread if needed!

timbo-tj
PROOP

2 years ago

Thanks

brody
EMPLOYEE

2 years ago

sounds good

Loading...