c17693fa-8f81-4f00-ad4d-8f231f687806

Are you using a metal region?

us-west1

I would tell you to analyze this problem further, the team will probably ask for more data because if it were a general problem there would be more cases here

Are you using something like uptime kuma to monitor this latency?

also, there's a metal us-west and a gcp one

I'm measuring the backend response time as stated earlier, that's why it caught my attention

users started complaining about random disconnections

and after checking the measurements, response time went from less than 1 second to almost 19 seconds per request

Some hours ago having 500 users at the same time still kept the 1 second time

Now there are just 60 and server is delaying up to 19 seconds per request

No logs, no errors, no anything. Tried even the good ol' reset just in case, even with a fresh start there is heavy lag now

gotcha, let's wait for a team/conductor to answer this thread

Okay, now all out of a sudden it went from 22-25 seconds per response down to 3-7 seconds, still laggy, but a drastic improvement

Again, no change at all in the code itself, in fact logged users increased, so it isn't userload either

This sounds to me like your database and backend service are in different regions. Can you please send screenshots of both? @Kuha

both say US WEst Oregon

Please send screenshots

Great, that rules out metal/nonmetal

If you have any quantifiable data, such as a grafana dashboard, please share that

Sadly only what I stated before

userload and average response times

can you please try to switch both services to the v2 runtime

where is that?}

within the service settings

found it

if this doesnt change anything we would have to recommend you setup tracing so that you can pinpoint where the "slow" is coming from

I'll be testing with the V2 then

never knew that was there

what does it change?

moves the workload from docker to podman

So far average response time went down to 1.6 seconds, nice

will keep updating just in case

sounds good, I'll leave this thread open, but just know that if you see an increase in latency you will need to add tracing to your app

Railway has no observability into what your code is or is not doing.

@silence @Brody how did u know they weren't running in runtime v2? your admin superpowers?

silence fail

yes, though i didnt look at anything the user couldn't see themselves, if you see a runtime selector set to legacy then you are on legacy, if you do not see a runtime selector at all, then you are on v2, and the selector isnt there because you cannot go back to legacy

when did the runtime change?

A long time ago, lol

As of 2024/06/04 (YYYY/MM/DD)

makes sense, I have it running since 2023 almost haha

It's @silent lol

@Medim

oh yeah

mb

s'all good

!s