Slow to connect replica

timbo-tj

PROOP

2 years ago

I am using Uptime Kuma to detect down services. Our game server has a few replicas. And for some reason it seems like one of the replicas are 'slow' to connect to. Using Chrome developer tools I can see that it will randomly take a very long time (1s+) to connect to the replica. I am pinging the health-check endpoint which returns how long the check took. My server responds with <10-20ms, but the request does not resolve for over 1s+.

It only randomly happens. I can keep pinging the health-check end point and I get a lot of very quick responses. But every 4-5 attempts hits the 'slow' replica. (I assume).

Project ID: 4c3b4b0e-006a-407e-90c7-9c3031cd622f

Service: 2077a108-22c6-4367-951d-41afafbfd450

Health-check endpoint is {service url}/health-check

Maybe this will resolve on its own soon, I will keep an eye on it. For now I have set the timeout to 2s so we aren't being flodded with alerts.

34 Replies

brody

EMPLOYEE

2 years ago

Are you using the edge proxy?

timbo-tj

PROOP

2 years ago

whats the edge proxy

brody

EMPLOYEE

2 years ago

a new home grown proxy that will replace railways current envoy proxy, it's meant as a faster (to say the least) replacement

brody

EMPLOYEE

2 years ago

and it's a simple toggle switch away

timbo-tj

PROOP

2 years ago

i dont know anything about it

timbo-tj

PROOP

2 years ago

i can try it out?

brody

EMPLOYEE

2 years ago

absolutely

brody

EMPLOYEE

2 years ago

please do

timbo-tj

PROOP

2 years ago

this happened right after i pushed some changes and forced a new build, which i know would suggest its probably something i did.. but..! the pattern doesnt make sense. at all. <:3HC_think:712416049236017152>

timbo-tj

PROOP

2 years ago

maybe i can log out the instance id on health check to see if its the same instnace that is responding slowly

brody

EMPLOYEE

2 years ago

that would be a very good idea, do you know the env var for the replica id?

timbo-tj

PROOP

2 years ago

yeah i think so