Problem with Redis after migrations (ECONNRESET)
rasmuslian
HOBBYOP

2 years ago

Hi!
I have a serivce with Medusa.js (Node), Redis and Postgres which I've had since september. I've had no problems with this service before, but after I migrated to the new databases I have gotten a problem where the service sometimes stops working (not immediately, but after a while, seemingly at random times). It doesn't crash but the API just times out or gets 404.

I've found the logs which I think are the culprit, but I still haven't found a solution yet. It might have something to do with Redis timing out or hitting memory limits perhaps? Just a guess from my part so far. Here are the logs:

33 Replies

rasmuslian
HOBBYOP

2 years ago

9b3bd973-ba5c-4ef3-9f43-32f499f7ba19


Anonymous
TRIAL

2 years ago

FYI I've started experiencing this recently as well, with a PG instance - did not used to have these issues before


rasmuslian
HOBBYOP

2 years ago

Hm, really strange. It really messes our product up at the moment


Anonymous
TRIAL

2 years ago

I'm not experiencing it on both my services, so could be an app layer issue but didn't use to happen 🤷‍♂️


brody
EMPLOYEE

2 years ago

often times this happens when you aren't closing connections and the idle timeout is reached


Anonymous
TRIAL

2 years ago

On my end my websockets are maybe suspicious, i'll take a look at some websocket settings


brody
EMPLOYEE

2 years ago

for postgres pooled clients specifically this is solved by setting the pool minimum to 0 so that all connections are released and marked as closed


rasmuslian
HOBBYOP

2 years ago

Where do I set that?


brody
EMPLOYEE

2 years ago

your issue looks to be with redis, either way you would need to reference the documentation for your database client


rasmuslian
HOBBYOP

2 years ago

Ah true, is there a similar pool minimum setting for Redis?


brody
EMPLOYEE

2 years ago

not sure, you would need to reference the documentation for your database client


rasmuslian
HOBBYOP

2 years ago

That is Medusa then you mean in my case?


brody
EMPLOYEE

2 years ago

medusa is not a database client, the redis npm package is


rasmuslian
HOBBYOP

2 years ago

Okey, will check there


Anonymous
TRIAL

2 years ago

Thank you for the help Brody!


Anonymous
TRIAL

2 years ago

If the connections to the DB are idle that means there are no DB operations? I should be having constant traffic 🤔 maybe I'm having some other underlying issue.

Or I am missunderstanding. Nonetheless trying to put min:0 and hoping for the best! 😄


brody
EMPLOYEE

2 years ago

what tech stack are you using?


Anonymous
TRIAL

2 years ago

nestjs with socketio and typeorm to connect to PG


Anonymous
TRIAL

2 years ago

I'm suspicious of socketio as well


brody
EMPLOYEE

2 years ago

does typeorm have a pool.min setting?


Anonymous
TRIAL

2 years ago

They have a poolSize which is max, but they also accept extra and pass it onto the underlying driver. So that should accept min I believe


brody
EMPLOYEE

2 years ago

whats the underlying driver in use?


Anonymous
TRIAL

2 years ago

pg and as you say that, realizing that maybe it doesn't have a min setting and I need to use the idleTimeoutMillis or allowExitOnIdle


brody
EMPLOYEE

2 years ago

allowExitOnIdle seems like what we want


Anonymous
TRIAL

2 years ago

Thank you for the help, I'll try it out!


brody
EMPLOYEE

2 years ago

let me know how that goes!


artisanvaultcode
PRO

2 years ago

the problem inside is the proxy with tcp protocol


brody
EMPLOYEE

2 years ago

sorry but the issue here does not lie with railway, they have internal monitoring for these kinds of things and nothing has been reported. these errors are due to how the client is handling connections


artisanvaultcode
PRO

2 years ago

WHat do you know!


Anonymous
TRIAL

2 years ago

@Rasmus Lian I have the same issue with medusa, only started since we migrated. Have you found a solution?


rasmuslian
HOBBYOP

2 years ago

@latrapo Yes I solved it by upgrading Medusa and Redis services I think (cache service and notification provider) and make sure the Redis config is correct. I think that was it, I am afk atm so cant give you more atm


Anonymous
TRIAL

2 years ago

Thanks! Will try doing that in the mean time 🙏


brody
EMPLOYEE

2 years ago

make sure you aren't keeping any idle connections around and are sure to close a connection when done with it


Loading...